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I. INTRODUCTION 


The source selection phase of government acquisitions involves many interrelated 
tasks. At the beginning, teams of experts determine the evaluation factors and significant 
subfactors that represent the key concerns of the procurement and incorporate those 
factors into a formal document which is called a Request for Proposal (RFP). Once 
proposals are received, experts evaluate each proposal against the standards provided in 
the RFP. If negotiated contracting is the selected method for the acquisition, contractors 
are allowed to revise their offers and discussions are conducted before the contract is 
awarded. Otherwise, evaluations are conducted without negotiations and the contract is 


awarded directly by the Source Selection Authority (SSA) based on the evaluations.! 


Even though objective factors and subfactors are used in the source selection 
phase, subjectivity is inevitable in government contracting. While the Federal Acquisition 
Regulation (FAR) does not provide any clear guidance on evaluating proposals, the 
source selection decision is usually made based on some evaluations “conducted using 
any rating method or a combination of methods, including color or adjectival ratings, 
numerical weights, and ordinal rankings.”’* Besides, proposals are evaluated by different 
personnel and conclusions may differ significantly, even for the same factor.? Therefore, 
considering the high complexity in the nature of government acquisitions, comparison of 
proposals and the source selection decision involve subjective judgmental evaluations, 


and these evaluations may be in conflict with fairness and the best value.* 


Considering the fact that resources available to governments are becoming 
scarcer, subjectivity should be reduced as much as possible in contracting, and agencies 


should find some ways to get the best value from any procurement. 


! Carl R. Templin and Ken R. Noffsinger, “An Assessment of the Role of Technical and Evaluation 
Factors,” International Journal of Purchasing and Materials Management 30 (1994); 38. 


2 Federal Acquisition Regulation (FAR) 15.305. 


3 John Cibinic and Ralph C. Nash, Formation of Government Contracts, 3rd ed. (Washington, DC: 
The George Washington University, 1998), 821. 


4 Templin and Noffsinger, “An Assessment,” 38. 
1 


As a solution to the problems stated above, quantitative methods and information 
technology tools can be utilized to augment objectivity in the source selection decision 
making. Quantitative methods and information technology tools can assist with decision- 
making by analyzing data and providing some solutions. Choosing an optimal solution 
among different alternatives using many evaluation criteria, performing risk assessments, 
and planning complicated operations are some advantages of information technology 
tools. “Implementation and use of these tools in a systematic method can help shift 
government purchasing from a tactical, reactive mode of operation to a more strategic, 


proactive mode of operation.’”> 


In this context, the aim of this study is to show the effects of the subjectivity 
involved in the government procurement processes and suggest some possible solutions. 
The authors develop and analyze multiple versions of numerical models based on the 
weighted sum method, which is one of the tools that can be used to solve multi-criteria 
decision making problems. These models are built in Microsoft Excel using data from a 


current procurement program. 


In this study, the USAF’s KC-X Tanker Replacement Program has been selected 
as the program to be used to frame the research questions. There are some specific 
reasons for choosing that program. First of all, an Air Force procurement program in 
which color ratings were used has been chosen to show the effects of subjectivity in 
government contracting because, in the authors’ perspective, the Air Force’s color rating 
method is one of the methods that is incapable of considering small differences between 
proposals and thus causes subjectivity to play a role during the decision process. In 
addition, a popular and a current program like the USAF’s KC-X program has been 
selected to be able to reach lots of different comments from many different viewpoints. 
Considering the confidentiality issues of past procurement programs that make it 
impossible to reach enough information, a current program provides much more publicly 
available information. After all these considerations, the authors came up with the idea of 


doing their study on the USAF’s KC-X program. 


5 Kathy L. Spainhower, “An Exploratory Study on the Strategic Use of Information Technology,” 
(master’s thesis, Air Force Institute of Technology, 1998), 7. 
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After doing some research about the KC-X program, it has been discovered that 
the two different proposals offered very different specifications and capabilities for the 
same program. This showed the difference between the two offerors’ viewpoints and 
made it clear that this program might be a good sample for this study because of the 


potential effects of subjectivity. 


Another factor that makes this choice even more interesting is the high likelihood 
of the contract award being protested. Experts of the defense industry were expecting a 
protest regardless of the winner. Since the proposed KC-X alternatives have very 
different strengths in capabilities, the results would not easily satisfy the offeror that loses 
the program award. As expected, after the authors started working on the KC-X program 
to show the subjectivity involved in source selection (mostly because of the color rating 
technique) and its effects on the source selection decision, the program was awarded to 
the Northrop Grumman-Airbus EADS team and Boeing immediately protested the 
program to the Government Accountability Office (GAO). 


In this study, detailed explanations about government contracting and decision- 
making support techniques are presented first to provide readers with some background 
information. After that, the USAF’s KC-X Tanker Replacement Program is examined in 
detail to demonstrate its special features and high importance. Also, this part provides the 
color rating data of both offerors that were found in publicly available sources and used 
in the models. After these explanatory chapters, the model construction chapter shows 
how the two versions of the model were built with some simple examples. In the chapter 
that discusses the analysis and results, it is shown with both versions of the model that the 
final source selection result of the KC-X Tanker Replacement Program could have been 
different if the USAF had used the weighted sum method as its source selection 
evaluation method. Then, optimization is built into the models to perform sensitivity 
analysis with respect to choices that were made when constructing the models. In the 
final part, the conclusions from each part of the study are summarized and 


recommendations for further research are presented to the readers. 
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I. ACQUISITION OVERVIEW 


A. INTRODUCTION 


Government acquisitions are to be conducted in compliance with many laws and 
regulations that require contracting agencies to ensure that all contractors are treated in an 
equal manner and that the government acquires the supplies or services for a fair and 


reasonable price.® 


In awarding U.S. government contracts, The Competition in Contract Act of 1984 
(CICA) requires ‘full and open competition’ to be provided by contracting officers. 
According to CICA, “full and open competition means that all responsible sources are 
permitted to submit sealed bids or competitive proposals on the procurement.”’ The 
importance of full and open competition for government is that if performed ideally, it 
results in quality services and products provided to the government at desirable costs. It 
has been experienced that, on average, 25% of the total cost can be saved by using 


competitive processes.8 


The source selection phase in government contracting involves multi-criteria 
decision making that has to deal with many critical factors, such as cost, technical 
capabilities, and past performance of contractors. The contracting processes used in 
government acquisitions have been built to respond to various government procurement 


needs by evaluating these factors. 


Under some requirements, contracting officers may use contracting procedures 
that provide no full and open competition and the justifications for awarding a contract 


without competition is stated in Federal Acquisition Regulation (FAR) 6.303. Based on 





© Templin and Noffsinger, “An Assessment,” 38. 


7 William Thybony, Government Contracting Based on the Acquisition Regulation (and the 
Competition in Contracting Act of 1984) (Thybony, Inc., 1985), 95. 


8 Ibid., 99. 


the circumstances, “such procurements may be made on either sole source basis or with 
limited competition.”? According to FAR 6.302, situations where full and open 


competition is not required are as follows: 


e “Only one responsible source and no other supplies or services will satisfy 
agency requirements 

e Unusual and compelling urgency 

e Industrial mobilization; engineering, developmental, or research 


capability; or expert services 


e International agreement 

° Authorized or required by statute 
e National security 

e Public interest” 


This chapter is mainly divided into two parts. The first part (Section B) provides 
general background information about government contracting. While both methods of 
government contracting, which are sealed bidding and contracting by negotiations, are 
explained during this part of the study, the main focus of this paper is on Contracting by 
Negotiations because it is the technique more commonly used in government contracting 


practices. 


The second part (Section C) provides information about Air Force source 
selection procedures because the authors are going to use a U.S. Air Force weapon 
system acquisition, namely the KC-X Tanker Replacement Program, as the study case 
and example to demonstrate the subjectivity involved in government source selection 


decisions. 
B. CONTRACTING METHODS 


Contracting officers are free to choose the method of contracting, but they are 
required to choose the best responsive method that meets all government requirements. !° 


Basically, the complexity of the acquisition and the need for discussion determine the 


9 Cibinic and Nash, Formation of Government Contracts, 280. 
10 Tbid., 312. 


methods of contract awarding. For instance, in non-complex acquisitions which do not 
require discussion, a contract can be awarded to the lowest priced offeror who has 
proposed to provide at least minimum technical requirements. On the other hand, 
complex acquisitions, for instance big weapon systems procurements, lead contracting 
officers to use other options that help to determine the offeror whose proposal would give 


more value to the government.!! 


In order to accomplish full and open competition, the following competitive 
procedures are used according to FAR 6.102: 
° Sealed bids 


° Competitive proposals 
e Combination of competitive procedures 
e Other competitive procedures 


In this study, the first three will be examined while stressing the attributes of only 


the sealed bidding and the competitive procedures. 
1. Sealed Bidding 


The purpose of sealed bidding is to encourage competition while avoiding fraud 
or favoritism.!2 While determining the procedures that will be used for a contract, 


according to FAR 6.401, contracting officers shall solicit sealed bids if: 


° “Time permits the solicitation, submission, and evaluation of sealed bids; 

e The award will be made on the basis of price and other price-related 
factors; 

e It is not necessary to conduct discussions with the responding offerors 
about their bids, 

e There is a reasonable expectation of receiving more than one sealed bid.” 


11 Spainhower, “An Exploratory Study,” 14. 


12 Noel W. Keyes, Government Contracts under the Federal Acquisition Regulation (St. Paul, MN: 
West Publishing Co., 1986), 153. 
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In sealed bidding, price and price-related factors play a great role in choosing the 
offeror. According to FAR 14.103-2, “an award is made to the responsible bidder whose 
bid is responsive to the terms of the invitation for bids and is most advantageous to the 
government, considering only price and the price related factors included in the 


invitation.” 


The price-related factors in FAR 14.201-8 include: 


e “Foreseeable costs or delays to the government resulting from such factors 
as differences in inspection, locations of supplies, and transportation. 

° Changes made, or requested by the bidder, in any of the provisions of the 
invitation for bids. 

e Advantages or disadvantages to the government that might result from 
making more than one award. 

e Federal, state, and local taxes. 

e Origin of supplies, and, if foreign, the application of the Buy American 


Act or any other prohibition on foreign purchases.” 


The evaluation process becomes more complex when there are more factors to be 
considered. Multiple items, indefinite quantities, extended performance periods, or price 
adjustment provisions make this evaluation process more complex and hard to manage. 
Additionally, decision making turns to its most difficult form when price-related factors 
are included in the evaluation formula.!3 In solicitation, the contracting agency must 
provide all price-related factors and evaluation methods in a clear manner so that all 


bidders can easily understand how their proposals will be evaluated. !4 


When two or more bids are equal after the evaluation, the contracting agency shall 


award the contract based on the following order of priority: 


e “Small business concerns that are also labor surplus area concerns, 
° Other small business concerns, 

e Other business concerns that are also labor surplus area concerns, 
e Other business concerns.”!5 


13 Cibinic and Nash, Formation of Government Contracts, 593. 
14 Tbid., 614. 
15 Thybony, Government Contracting, 147. 


If there are still equal bids after going through these priority steps, the contract is awarded 


by a drawing witnessed by bidding contractors if possible.!© 
Zz Contracting by Negotiation 


Negotiated contracting includes discussions with contractors and it usually gives 
an opportunity to contractors to revise their offers before the contract is awarded. 
According to FAR 15.302, “the objective of source selection in negotiated contracting is 
to select the proposal that represents the best value.” Best value basically means the 


greatest benefit that the government can get from an acquisition. 


In order to get more value through flexibility obtained by negotiations, statutory 
changes require discussions while awarding a contract, more detailed information about 
factor and subfactors in solicitations, limitations of competitive range, and detailed 
debriefings of offerors. Also, CICA orders competitive procedures or a combination of 


procedures, whichever are suitable, to be used.!7 


In FAR 15.002, in order to get the benefits of the flexibility of negotiation, types 
of negotiation, which are competitive and sole source acquisitions, are described in a way 
that aims to reduce complexity due to complicated solicitations that involve unnecessary 


large numbers of evaluation factors and subfactors.!8 


Contracting agencies can use any process or a combination of processes to get the 
best value from procurement. These processes basically involve “trade-off” and “lowest 


price technically acceptable source selection.”!9 


The more detailed background information about contracting by negotiation 
provided after this point will be divided into three sublevels, which are: selection 
processes, evaluation factors and subfactors and their relative importance, and evaluation 


of factors. 





16 Thybony, Government Contracting, 148. 

EY Ibid 313; 

18 Cibinic and Nash, Formation of Government Contracts, 710. 
19 FAR 15.101. 


a. Selection Processes 


Before soliciting proposals, contracting agencies shall determine the 
strategy that will be followed to make the source selection decision. The strategy that is 


used by agencies is also called “decisional rule.’’2° 


A solicitation, at least, should state that “evaluation factors other than cost 


or price, when combined, are- 


e significantly more important than cost or price, 
e approximately equal in importance to cost or price; or 
e significantly less important than cost or price.”2! 


The expected outcome of any strategy is to give the government the best 
value from any procurement. Best value is the greatest benefit that can be obtained in 


response to the requirement. 
In FAR 15.101, “best value” is explained as follows: 


An agency can obtain best value in negotiated acquisitions by using any 
one or a combination of source selection approaches. In different types of 
acquisitions, the relative importance of cost or price may vary. For 
example, in acquisitions where the requirement is clearly definable and the 
risk of unsuccessful contract performance is minimal, cost or price may 
play a dominant role in source selection. The less definitive the 
requirement, the more development work required, or the greater the 
performance risk, the more technical or past performance considerations 
may play a dominant role in source selection. 


(1) Trade-off Process. Contracting by negotiation allows 
contracting agencies to make trade-offs between cost and other factors.” This process 
aims to find the best value proposal by comparing the differences between technical 


specifications and costs and to justify paying more than the lowest price. Therefore, 


20 Cibinic and Nash, Formation of Government Contracts, 712. 
7! bid. 
>? Tbid., 710. 
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contracting agencies may not always select the lowest-priced bidder. A contract may be 
awarded to a higher-priced bidder if the justification is that of better quality of the 


proposed product or service.” 
This is explained in FAR 15.101-1 as follows: 


(a) “A tradeoff process is appropriate when it may be in the best 
interest of the Government to consider award to other than 
the lowest priced offeror or other than the highest technically 
rated offeror. 


(c) This process permits tradeoffs among cost or price and non- 
cost factors and allows the Government to accept other than 
the lowest priced proposal. The perceived benefits of the 
higher priced proposal shall merit the additional cost, and the 
rationale for tradeoffs must be documented.” 


(2) Lowest Price Technically Acceptable Source Selection Process. 
In this process, the best value concept is accomplished when the strategy is chosen, not 
after or during evaluation of the proposals.?4 In other words, it is certain that the offeror 
that proposes a technically acceptable product or service with the lowest price gives the 
possible best value to the government in this process. FAR 15.101-2 describes the 
process as follows: 
(a) The lowest price technically acceptable source selection process is 


appropriate when best value is expected to result from selection of the 
technically acceptable proposal with the lowest evaluated price. 


When evaluating proposals that meet or exceed requirements for 
non-cost factors based on price, non-cost factors have equal importance. Therefore, if a 
proposal does not meet any of the non-cost factors, the offeror with that deficient 


proposed product or service loses the contract.2> 





23 John R. Trumm, A Decision Analysis Tool for the Source Selection Process (Air Force Institute of 
Technology, 2006), 18. 


24 Cibinic and Nash, Formation of Government Contracts, 715. 


25 Tbid., 716. 
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(3) Combination of Trade-off and Low Price Acceptable 
Processes. Although FAR does not provide any explanation in regards to combined 
processes, in part 15.101 it allows them to be used. In this process, while some factors 
are evaluated based on a “go, no-go” basis, others are evaluated on their relative values. 
Otherwise, as another strategy, first all factors are evaluated based on a “go, no-go” basis 


and the proposals that can pass this first step are evaluated on their relative merit.2° 


b. Evaluation Factors and Subfactors and Their Relative 
Importance 


Evaluation factors and subfactors are milestones in a contract that describe 
what is significant for the procurement needs. These factors and their relative importance 


should be clearly described in solicitation to be completely understood by offerors. 


According to FAR 15.304, “these factors and subfactors must 


e represent the key areas of importance and emphasis to be 
considered in the source selection decision; and 


® support meaningful comparison and discrimination between and 
among competing proposals.” 


According to FAR 15.304, contracting officers are free to choose these 
factors and determine their relative importance. However, there are some requirements to 
be followed related to these factors: 

e In all government contracts, cost should be evaluated. 


° One or more non-cost factors should be evaluated to ensure the 
quality of products or services. 


e Past performance shall be evaluated unless the contracting officer 
documents that it is not an appropriate evaluation factor. 


The relative importance of the evaluation factors should be stated in two 
steps in the request for proposal (RFP). In the first step, the relationship of cost/price to 
the non-price factors should be pointed out and then, as a second step, the relative 
importance of the factors and subfactors in the non-cost or price areas should be 


described. 


26 Cibinic and Nash, Formation of Government Contracts, 718. 
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(1) Relationship of Cost/Price to Other Factors. An RFP should, at 
least, involve a statement indicating the decisional logic that will be used to make the 
source selection decision. It is essential for establishing competition to inform each 
offeror whether the award will be made based on the lowest-price acceptable proposal 


basis or trade-off basis. 27 


Variable relationships and fixed relationships can be used to 
determine the importance of the factors. A variable relationship is usually preferred when 
the cost to government will vary depending on the other factors. Also, variable 


relationships among other factors are possible. 28 


(2) Relative Importance of Other Factors. There are a lot of 


different methods to determine the relative importance of factors other than cost or price. 


One of the simplest ways to disclose the relative importance of 
factors is to list the factors that will be used in the evaluation. With this method, no 
explanation of differences between factors is provided and the offerors may assume that 
all evaluation factors have approximately equal importance in determining the winning 


proposal.29 


Another way to indicate the relative importance is to list the factors 
in descending order based on their importance without assigning a weight to each of 
them. This method is useful when there is “a reasonable downward progression of 
evaluation weights.”3° On the other hand, this method is satisfactory when the difference 
in the importance of two factors following one another is small. Besides, if a factor has 


more importance compared to others, this should be disclosed in the RFP.3! 


27 Cibinic and Nash, Formation of Government Contracts, 748. 
28 Ibid. 
29 Tbid., 751. 
30 Ibid. 
3! Ibid. 
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c. Evaluation of Factors and Subfactors 


According to FAR 15.305, “evaluations may be conducted using any 


rating method or combination of methods, including color or adjectival ratings, numerical 


weights, and ordinal rankings.” 


Additionally, DoD Directive 4105.62 states: 


There is no prescribed methodology for rating. Past practices include color 
coding, numerical, and plus or minus checks. The important thing is not 
the rating methodology but the consistency with which it is applied to 
elements of proposals and among proposals to ensure a though and fair 
evaluation. Evaluators must be well rounded in their field of expertise and 
be able to apply mature professional judgment....evaluators must support 
the rating assigned with a concise narrative that addresses strengths, 


weaknesses, and risks in the proposal.32 


As advised by these regulations, contracting agencies can use words or 


symbols to describe the compatibility levels of each evaluation factor and this method is 


called adjectival ranking. For instance, the Air Force uses a descriptive adjective system, 


a color coding system, and a symbol system.33 Since the Air Force source selection rating 


system is one the most significant cornerstones in this study, more focus will be given to 


it and explanations about it will be provided throughout the study. 


Another method for evaluating the factors is to use numeric point scores or 


percentages for each of them. This method may be a good guide for source selection 


authorities while selecting the best offeror, but it is warned that it should not be used as a 


primary tool in the decision-making process. It is stated that: 


Numerical point scores, when used for proposal evaluation, are useful as 
guides to intelligent decision making, but are not themselves controlling in 
determining award, since these scores can only reflect the disparate, 
subjective, and objective judgments of the evaluators. Whether a given 
point spread between competing offers indicates the significant superiority 
of one proposal over another depends on the facts and circumstances of 


32 Spainhower, “An Exploratory Study,” 22. 


33 Tbid. 
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each procurement, and while technical scores must of course be 
considered by Source Selection Authorities (SSAs), such officials are not 
bound thereby.>4 


Table 1 below shows a combination of adjectival and numerical 






































systems:35 
. Definition / Evaluation 
Scores Rating 
Innovative, comprehensive and complete in all details, 
10 Excellent meets all requirements and objectives without “gold 
plating” 
Substantial response in clearly definable detail, meets 
9 Very Good ty 
all critical requirements 
7 Average Generally meets minimum requirements 
Lack of essential information to substantiate data 
6 Poor 
presented 
; Lack of understanding of requirements or omissions in 
5 Unsatisfactory 
major areas 
0 No data 
Table 1. | Sample Cross Reference of Numerical and Adjectival Scoring Systems. 


C. SOURCE SELECTION IN UNITED STATES AIR FORCE 


The objective of source selection in the U.S. Air Force is to select the offeror 
which proposes the best value to the government. In major weapon system procurements, 
the Air Force uses contracting by negotiation procedures to encourage discussions and 


negotiations to reach the best value. The confidence in an offeror’s ability to meet all 


34 Spainhower, “An Exploratory Study,” 23. 
35 Tbid. 
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requirements perfectly is the key in a best value continuum, and this confidence may lead 
SSA to award a contract to an offeror which is not the one offering the lowest cost to the 
government. On the other hand, openness and early industry participation are the main 


points associated with the relations with industry in the acquisitions of the Air Force.*® 
i, Categories of Air Force Acquisitions 


There are three categories of Air Force source selection based on the dollar value 


and complexity of the acquisition. The categories are as follows: 


e Basic source selection 
e Median source selection 
e Agency source selection 


Basic source selection is the easiest one among all three types and the associated 
procedures are simple, direct and minimal. For non-information technology negotiated 
acquisitions, the threshold is $10 million and for information technology efforts, the cost 
range should be less than $15 million per one year and less than $30 million for the 


whole program.37 


Median source selection is used for moderately complex source selections and 
needs a more structured approach than basic source selections. Cost/price, past 
performance, mission capability, and proposal risk are always used as factors in this type 
of source selection. For non-information technology negotiated acquisitions, the 
minimum threshold is $10 million and the maximum threshold is $100 million. On the 
other hand, for information technology efforts, “these procedures apply to Non-Major 
Automated Information Systems (MAIS) that are equal to or greater than $15 million in a 
fiscal year or are equal to or greater than $30 million for the total program but less than 


$120 million.”38 





36 Air Force Source Selection Procedures Guide, 3, https://www.afmce-mil.wpafb.af.mil/HQ- 
AFMC/PK/pkp/polvault/guides/sspguide.doc (accessed December 20, 2007). 


37 Thid., 5. 
38 Tbid., 6. 
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Agency source selection procedures are used for the most complex acquisitions 
that need a more structured approach than median source selections. Under the mission 
capability factor, there are several subfactors and elements to be evaluated if necessary. 
For non-information technology negotiated acquisitions, the minimum threshold is $100 
million. For information technology acquisitions, the minimum threshold is $120 million 


to use agency source selection procedures.>9 
2. Steps in Air Force Source Selection 


All these three types of source selection procedures are composed of the same 
three steps, which are: 
1. Pre-Solicitation activities 
2. Evaluation activities 


3. Award activities 


In this study, because the authors are going to use a major weapon system 
acquisition as an example, the cost of which exceeds billions of dollars, the agency 


source selection procedure will be the focus of discussion. 
a. Pre-Solicitation Activities 


After determining the requirements, early participation of the industry and 
use of oral presentations instead of written statements are recommended in this stage of 
source selection. The other significant activity is to identify high-risk areas and determine 
discriminators for the selection. Determining these important factors makes it possible to 
build the basis for award, evaluation criteria, and evaluation factors which are announced 


in the source selection plan. 


(1) Evaluation Factors. The following four factors are used in Air 
Force source selection procedures: 
e Mission capability 


e Proposal risk 


39 Air Force Source Selection Procedures Guide, 7. 
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e Past performance 


e Price or cost evaluation factor 


The mission capability factor focuses on the technical requirements 
that have significant value in the source selection decision. If subfactors under this factor 
are to be used, the number of subfactors can not exceed six.49 The need for using 
subfactors and elements are determined by performing a formal risk assessment. Keeping 
the numbers of subfactors and elements low helps the evaluation team make a meaningful 


and complete evaluation. +! 


The proposal risk factor focuses on the risks an offeror puts on the 
program’s significant goals, which are cost, schedule and performance. This factor is not 


mandatory for acquisitions of $10 million or below.*2 


The past performance factor focuses on the confidence that an 
offeror provides to the government showing the offeror’s ability to supply products or 
services while meeting all requirements stated in the solicitation. It may be the most 
important factor or at least as important as the most important non-cost factor in the 


evaluation.*? No subfactor is typically assigned to this factor like the price/cost factor.*4 


Price/cost evaluation focuses only on reasonableness and realism 
aspects of the price/cost proposed by an offeror. Therefore, the data requested from an 
offeror to perform this evaluation should be limited to the amount required to perform the 


reasonableness and realism assessment.*> 


40 Air Force Mandatory Source Selection Procedures 4.4.1.1, 
http://farsite.hill.af.mil/reghtml/regs/far2afmefars/af_afmc/affars/5315.htm#P41_ 1723 (accessed January 
14, 2007). 





41 Air Force Source Selection Procedures Guide, 43. 
42 Air Force Mandatory Source Selection Procedures 4.4.1.2. 
43 Air Force Source Selection Procedures Guide, 42. 


44 Alexander R. Slate, “Best Value Source Selection: The Air Force Approach, Part 1,” Defense AT&L 
(September-October 2004); 52. 


45 Air Force Source Selection Procedures Guide, 43. 
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After determining all factors, subfactors, and elements, as well as 


their relative importance, a formal request for proposal is issued and released. 
b. Evaluation Factor Assessment Activities 


The goal of source selection evaluation in Air Force acquisitions is to 
provide the best value to the government. Proposals given by offerors can only be 
evaluated based on the factors, subfactors and elements, if applicable, included in the 


RFP. 


(1) Mission Capability Assessment. The mission capability 
assessment focuses on the deficiencies and strengths in the offeror’s proposal. If there are 
subfactors established under this factor, they should be rated at a subfactor level and an 
overall factor rating is not assigned. This factor is rated using colors and each of the 
mission capability subfactors should get a color coding based on the performance 
capability an offeror proposes.*© Table 2 depicts the color ratings and their descriptions 


used in this evaluation. 


MISSION CAPABILITY COLOR RATINGS 


Rating Description 





Exceeds specified minimum performance or capability 
requirements in a way beneficial to the government; 
proposal must have one or more strengths and no 
deficiencies to receive a blue rating. 


Exceptional 





Meets specified minimum performance or capability 
requirements delineated in the RFP; proposal rated green 
must have no deficiencies but may have one or more 
strengths. 


Acceptable 





Does not clearly meet some specified minimum performance 
Yellow Marginal or capability requirements delineated in the RFP, but any 
such uncertainty is correctable. 











46 Air Force Mandatory Source Selection Procedures 5.5.1. 
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Fails to meet specified minimum performance or capability 
requirements; proposal has one or more deficiencies. 


Unacceptable Proposals with an unacceptable rating are not awardable. 





Table 2. Mission Capability Color Ratings. 


A yellow/marginal rating should be eliminated through 
information exchanges and at the end of the final evaluation, it should be rare.4” If a red 


rating remains, that proposal can not be awarded the contract. 


Along with color ratings, SSA should be informed with narrative 
assessments, and those assessments should not compare offerors, they should only 


evaluate each offeror’s proposal against requirements established in the RFP.*8 


(2) Proposal Risk Assessment. Proposal risk assessment focuses 
on the weaknesses in the proposed approach. It should be evaluated and rated at the 
subfactor level against the mission capability subfactors, and an overall rating is not 


assigned.49 Table 3 demonstrates the ratings used in this evaluation. 


PROPOSAL RISK RATINGS 


Rating Description 





Likely to cause significant disruption of schedule, increased cost, or degradation of 
High performance. Risk may be unacceptable even with special contractor emphasis and 
close government monitoring. 





Can potentially cause disruption of schedule, increased cost, or degradation of 
Moderate | performance. Special contractor emphasis and close government monitoring will likely 
be able to overcome any difficulties. 


Has little potential to cause disruption of schedule, increased cost, or degradation of 
performance. Normal contractor effort and normal government monitoring will likely be 
able to overcome any difficulties. 








Table 3. Proposal Risk Ratings. 


47 Air Force Mandatory Source Selection Procedures 5.5.1. 
48 Air Force Source Selection Procedures Guide, 46. 


49 Air Force Mandatory Source Selection Procedures 5.5.2. 
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(3) Past Performance Assessment. Past performance evaluation 
demonstrates the government’s confidence in an offeror’s ability to meet all requirements 
established in the RFP. The most critical aspects of the information obtained about the 


offeror’s past performance are recency and relevancy. In agency source selection, past 
performance should be evaluated by a Performance Risk Assessment Group (PRAG).99 

A confidence rating is assigned to each offeror to assess recent past 
performance, with focus on the performance related to the mission capability subfactors and 


cost or price.>! The six confidence assessment ratings are seen in Table 4. 


PERFORMANCE CONFIDENCE ASSESSMENTS 


Rating Description 





High Confidence Based on the offeror’s performance record, the government has high 
confidence the offeror will successfully perform with the required effort. 





Based on the offeror’s performance record, the government has significant 


Significant Confid 
ee ag a eT confidence the offeror will successfully perform with the required effort. 





Based on the offeror’s performance record, the government has confidence 
Satisfactory Confidence | the offeror will successfully perform the required effort. Normal contractor 
emphasis should preclude any problems. 





Uinlnown Confidence. | 2 performance record is identifiable. 


Based on the offeror’s performance record, substantial doubt exists that the 


Little Confidence : f 
offeror will successfully perform the required effort. 





No Conkdeice Based on the offeror’s performance record, extreme doubt exists that the 
offeror will successfully perform the required effort. 








Table 4. Performance Confidence Assessments. 


(4) Cost/Price Assessment. Cost/price assessment focuses on the 
fairness and realism of the cost/price data provided. A risk rating should be assigned to 
this factor to assess the risk involved to the proposed cost. Table 5 depicts the ratings 


used in the cost/price risk assessment. 


50 Air Force Source Selection Procedures Guide, 45. 
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COST/PRICE RISK RATINGS 


Rating Description 





Likely to cause significant disruption of schedule, increased cost, or degradation of 
High performance. Risk may be unacceptable even with special contractor emphasis and close 
government monitoring. 





Can potentially cause disruption of schedule, increased cost, or degradation of 
Moderate | performance. Special contractor emphasis and close government monitoring will likely 
be able to overcome any difficulties. 





Has little potential to cause disruption of schedule, increased cost, or degradation of 
performance. Normal contractor effort and normal government monitoring will likely be 
able to overcome any difficulties. 








Table 5. — Cost/Price Risk Ratings. 


Before awarding the contract, it may be necessary to exchange 
some information with offerors, and these information exchanges are composed of 
“clarifications”, “communications”, and “discussions.” After finishing these activities, 


final proposals are evaluated by the evaluation team. 
c. Award Activities 


After proposals are received, an evaluation team that consists of many 
experts performs a very detailed assessment and reaches the final color/adjectival ratings 
and narrative assessments for each offer. Before SSA makes the source selection 
decision, the evaluation team gives him a briefing containing all these assessments. This 


briefing includes the strengths, deficiencies, and weaknesses an offeror has in its proposal 


as the narrative assessments. The SSA evaluates all color/adjectival ratings and narrative 
assessments provided by the evaluation team and awards the contract to the offeror that 


proposes the best value to the government. 


After announcing the results and the winner, a debriefing that provides all 


evaluations with required explanations is presented to each offeror.53 


52 Air Force Source Selection Procedures Guide, 49. 
53 Tbid., 50. 
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D. SUMMARY 


In this chapter, a general overview of acquisition processes is provided. First, 
contracting methods are discussed and then evaluation and selection processes are 
defined briefly. In the second part of this chapter, because this project will focus on Air 
Force’s color rating method, Air Force source selection is examined in greater detail in 
terms of categories of Air Force acquisitions, steps in Air Force source selection, 
evaluation activities, and award activities. The next chapter will discuss the decision- 
making support techniques that allow a final assessment to be reached from the 


individual assessments explained in this chapter. 
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Hil. DECISION MAKING SUPPORT TECHNIQUES OVERVIEW 


A. INTRODUCTION 


Decision makers are trying to make more accurate decisions and their only assets 
in doing so are their personal experience and the available information about the 
situation. For accurate decisions, decision makers need relevant and correct information, 
and they need to analyze it correctly to come up with a beneficial final decision. With the 
help of information technology, which brings a lot of support techniques to help them 


make better analysis, decision makers can make better decisions. 


In this part of the authors’ study, they will give some background information 
about the available techniques to solve a multi-criteria problem because the example 
study case (the Air Force’s KC-X program) in this thesis is a major weapon system 
procurement and has more than one criterion to consider while trying to come up with the 
best value product for the government. In general, decision makers have two different 
choices to use to solve multi-criteria problems. A decision maker can decide on using a 
multi-objective optimization or a multi-criteria decision making technique. Under multi- 
criteria decision making, there are many different techniques available, but the authors 
choose the most popular eight methods to provide background information about. Also, 
there are many decision support software programs in the literature. These software 
programs are supporting tools that use one or some of the techniques, but the decision 
maker does not need to know anything about the techniques to use these programs. The 
authors will talk about two of them that best fit their needs. The authors will give a brief 
summary of each technique and as a conclusion they will make the determination about 
what technique they will use in their study. However, the aim of this chapter is not to 
provide guidance about choosing the appropriate technique where it supports the most 
accurate results for that special condition. Also, in the literature of techniques, it was 
stated that there are different conditions which make one of the techniques the most 
appropriate for that condition. Most of the literature studies do not provide the conditions 
that make the techniques the most appropriate ones for that specific condition. 
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B. OVERVIEW 


There are some different alternative techniques to help and support decision 
makers when they need to make a decision in real-life situations. Real-life situations 
consist of multiple criteria which often conflict with each other. In a multiple criteria 
decision making environment, there are two main options of techniques to choose from 
for a decision maker. A decision maker can choose any of the techniques under the multi- 
criteria decision-making umbrella or any of the optimization techniques, or otherwise any 
of the software under the decision support system or expert systems umbrella can be 
chosen to support the decision whenever a decision maker does not want to use any of the 


techniques. 


The chart below in Figure 1 is the summary of options under a hierarchical order 


for a decision maker to support his or her decisions for multi-criteria situations. 


MULTI-CRITERIA 
DECISION MAKING 
Multi-Objective Multi-Criteria 
Optimization Decision Making 
Methods 
WSM AHP Outranking TOPSIS CP MAUT 
or Methods 


WPM | 


PROMETHEE ELECTRE 


Figure 1. | Hierarchical Order for Multi-criteria Decision Making. 
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1. Multi-Objective Optimization 


The basic constrained single objective optimization problem is formulated as 


follows: 


Min J=f(X) 
h(X)=0 
g(X)<0 
DY. sox Sx 


S.t. 


where J is the objective function and J equals to a real valued function f(X), X is a vector 
of n real independent variables called decision variables, h(X) and g(X) are representing 
the feasible set for X (h represents the set of equality constraint functions and g represents 
the set of inequality constraint functions of X), and X,<=X<=Xy represents the lower and 


upper simple boundaries on feasible values of X. 


There are several different methods which can be used for solving optimization 
problems like the simplex method for linear constrained problems, sequential quadratic 
programming (SQP) for nonlinear constrained problems, and branch-and-bound 
techniques for discrete variables depending on the types of objectives, constraints and 


variables.54 


However, real-world applications often imply multiple objectives. For example, in 
product-design optimization, the cost and the quality of products are two conflicting 
objectives.5>5 Another example from real engineering design decisions typically involve 
both technical (i.e., maximize performance) and economic (i.e., minimize cost) 
considerations at the same time. In general, performance and cost are also opposing 
objectives, like quality and cost, and performance can only be increased by increasing 


cost. In these cases, the optimization problem becomes multi-objective.°° 


54 Robert A. Wolf, “Multi-Objective Collaborative Optimization of Systems of Systems,” (master’s 
thesis, Massachusetts Institute of Technology, 2005), 24. 


55Dirk Buche, Peter Stoll, and Petros Koumoutsakos, “An Evolutionary Algorithm for Multi-Objective 
Optimization of Combustion Processes,” Center for Turbulence Research Annual Research Briefs (2001); 
231, http://ctr.stanford.edu/ResBriefs0 1/bueche.pdf (accessed December 25, 2007). 


56 Wolf, “Multi-Objective Collaborative,” 24. 
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The typical multi-objective formulation is given as follows: 
Mi 
© I= F(X) =[A(X) AX) A] 
s.t. h(x) =0 
g(X)<0 
Xe aXE Xx, 


D.V. 


where J and equally f(X) is a function (vector) of n different objective functions at 
solution X. The solution to the multi-objective optimization problem is a set of Pareto 
points. Pareto solutions are those for which improvement in one objective can only occur 
with the worsening of at least one other objective. Thus, instead of a unique solution to 
the problem as in a single objective optimization, the solution to a multi-objective 
problem is a set of Pareto points. While working on multiple objectives, generally a 
single solution that optimizes all objectives simultaneously does not exist. There are 
multiple solutions that are "optimal" with respect to the different objective functions. 
Generally, the best solutions are the ones that represent a compromise between the 


various objectives.>7 


As a multi-objective problem creates multiple solutions, in order to come up with 
a single preferred solution, some method must be chosen to decide the best compromise 
which maximizes the utility function below. With this equation, multiple objective values 


can be combined into a single utility. 


ul f(X)J=ul A(X). A(X) 6,0) 


The methods to reach a single preferred solution can be divided into four primary 


classes:58 
e No preference methods (the distance to the ideal point is minimized-no 
input from decision maker) 
e A priori methods (decision makers define the utility function before any 


analysis) 


57 Wolf, “Multi-Objective Collaborative,” 25. 
58 Ibid. 
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e Iterative methods (identify decision maker preferences progressively as 
the design analysis progresses) 


° Posterior methods (identify the Pareto-optimal frontier and then allow the 
decision maker to determine the best compromise solution) 


2. Multi-Criteria Decision Making (MCDM) Methods 


Multi-criteria decision making is a well-known part of the decision making 
process and is an important part of the subject of operations research when dealing with 
more than one decision criteria to make a decision. Conflicts and incomparability 
(because of not being measurable) of some criteria are the problems that these techniques 
are mostly facing. There are several different methods under MCDM’s umbrella. Each 
method has its own characteristics and, therefore, each method can be the best method 
when the conditions suit its characteristics. However, in this study, as stated before, the 
aim is not to provide guidance for picking up the best technique for that special occasion 
(condition) or a list of best occasions for each technique. Likewise, most of the literature 
studies reviewed by the authors simply provide reasons why they think that the technique 
they choose fits the best for them. 


a. Weighted Sum Method (WSM) and Weighted Product Method 
(WPM) 


The WSM is one of the most popular and commonly used approaches, 
especially for single dimensional problems. When there are M alternatives and N criteria, 


in this method, the best alternative can be found by the following expression;>? 


N 
P 
Avs = Max,493,.M Sam, 
j=l 


where A*wsy is the weighted sum of the best alternative, aj is the actual value of hi 


h 


alternative for the j”' criterion, and w; is the weight of relative importance of the id 


criterion. The final value of each alternative is equal to the sum of values for each 


59 §. D. Pohekar and M. Ramachandran, “Application of Multi-Criteria Decision Making to 
Sustainable Energy Planning-A Review,” Renewable and Sustainable Energy Reviews 8 (2004): 368, 
http://web.nuu.edu.tw/~ctchen/download/96_1/m/96-05.pdf (accessed December 25, 2007). 
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criterion. This simple method can have problems when it is used for multi-dimensional 
problems and there are different units; it is meaningless to try to sum these results in 


different units.6° 


Since there is more than one criterion and different relative importance for 
each criterion, weights are used to capture the relative importance of the different criteria 
to the decision maker. This given relative importance of the criteria provides the model 
with tradeoffs, and with the help of weights, the model allocates their contribution to the 
overall score. In a scoring model, the weighted sum of the scores represents the project's 


overall value (score). 


While using this technique, it is necessary to first draw a hierarchy tree of 
objectives/factors and then start from the bottom level. Factors can have some subfactors 
and subfactors can have elements. Therefore, an overall score will be calculated for each 
subfactor group under a factor. Whenever an overall value is gotten from a sublevel, the 
procedure is started from the beginning for the new level. After getting an overall value 
(score) for the alternative, the same procedure is implemented for the next alternative. 
After getting an overall score for each alternative, the best option will be the alternative 


with the highest score for the decision maker’s preferences. 


The weighted sum method, which is also called multi-criteria scoring or 
the congruence model in different literary sources, is a basic and well-known technique 
that is used in different subject areas with sometimes different names. As it is used in 
MCDM (with three different names - weighted sum method, multi-criteria scoring or 
congruence model), it is also commonly used in economics under 'cost-benefit analysis 
for measure of effectiveness’ calculations. This technique is also used in statistics and 
mathematics as the weighted average (A method of computing a kind of arithmetic mean 
of a set of numbers in which some elements of the set carry more importance (weight) 


than others).©! 


60 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making.” 


61 Cliff T. Ragsdale, Spreadsheet Modeling and Decision Analysis: A Practical Introduction to 
Management Science (Mason: South-Western Learning, 2004), 806. 
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A very similar method to WSM is the weighted product method (WPM). 
The main difference between these two methods is that in WSM, one adds the values for 
each criterion to get the final value, while in WPM, multiplication of these values take 
place instead of addition. In WPM, each alternative is compared with others by 
multiplying the ratios created by each criterion. When trying to compare the alternatives 
by using WPM, one takes Ax and A; as two of the alternatives and solves the following 


expression;®2 


wi 


R(Ay/A,) => (ay ay) 


=l 


au 


where N shows the number of criteria, ajj and a ,; are the actual value of alternative K and 
L for the j” criterion, and w; is the weight of the relative importance of 7" criterion. When 
R(Ax/A_) is greater than 1, Ax is more preferable than A; (in maximization problems). 
The overall best alternative is the one which is more preferable than every other 


alternative.®3 
b. Analytical Hierarchy Process (AHP) 


The analytical hierarchy process (AHP), developed by Thomas L. Saaty at 
the Wharton Business School, provides a tool to decision makers to analyze decisions 
hierarchically with the overall goal of the decision at the top of the model, strategic 
objectives in the higher levels, and sub-objectives with evaluation criteria at the bottom.®* 
The hierarchy is fairly simple but the mathematics is a little complex. To start the 
process, a goal is broken down into objectives and sub-objectives, as shown in Figure 2, 


until each sub-objective can be defined using one or more measurable criteria. 





62 Ragsdale, Spreadsheet Modeling, 369. 
63 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 369. 


64 Michael C. Weinlein, “Funding for First Responders from a Threat and Prevention Approach,” 
(master’s thesis, Naval Postgraduate School, 2004), 22. 
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Goal 


Objective Objective Objective 
Sub Sub Sub Sub Sub Sub 
objective objective objective objective objective objective 


Figure 2. | AHP’s Hierarchical Approach. 


Once the hierarchy is established, weights are assigned to every branch in 
the tree. To do this, analysis starts at the bottom of the tree. Starting with each sub- 
objective, the measurable criteria for that sub-objective are compared using a pair-wise 
comparison method such that each criterion is compared against every other criterion, 
including itself. This creates a matrix of values that indicates the relative importance of 


every criterion versus every other criterion under a sub-objective. This matrix 


a, A a, 

Ay, Ay" Ag, 
AS om 5 

ayy ano Gan 


is created by putting a numerical value resulting from the result of the pair-wise 
comparison of criterion i with criterion j into the position aj, and its inverse into position 
aj, to obtain all of the pair-wise comparisons performed for a given sub-objective.© The 
values that are used as entries in this matrix, and their verbal descriptions, are given in 


Table 6. 


65 Weinlein, “Funding for First Responders.” 
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Relative importance 
Equally important 
Moderately more important 


Freely more important 





Table 6. | Fundamental Scale for Pair-Wise Comparisons. 


Once the matrix is determined, the values within each column of the 
matrix are then normalized so the entries within each column sum to one. Then, the 
values within each row are averaged to achieve a normalized vector of weights for the 
criteria (the values of this vector sum to one). This vector of weights can now be used to 
evaluate alternatives by summarizing the criteria values of each alternative into a single 
number in a similar way as the weighted sum method. This process is repeated for each 
level of the hierarchy so the evaluation of each alternative can be summarized in a single 
number for final comparison. The alternative with the highest final value should be taken 


as the best alternative. 


AHP also includes the calculation of an inconsistency index. This allows 
the decision maker to determine if his or her choices of pair-wise comparison scores are 
consistent. Whenever these choices are consistent, it means that the final decision is made 
well.66 A numerical example of the analytical hierarchy process can be found in 


Weinlein, pp. 22-24. 
c. Outranking Methods 


When there are not many alternatives to be evaluated in a complex 
problem, which has many criteria to be considered and many participants involved, 


outranking methods can be used by decision makers. 


66 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 369. 
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If the following two conditions are met, it is defined that the action a, 


outranks the action ai: 


e “a, 1s at least as good as a; with respect to a major subset of the 
criteria, 
° a, is not too bad relative to a; with respect to the remaining 


criteria.”’67 


The outranking methods demonstrate the dominance of one alternative 
over another by using ordinal and descriptive information. An important feature they 
have is that they provide limited preference ranking, not a full ranking. They do not 
require a certain utility function that shows the exact preference structure of decision 
makers. They just need enough information to indicate that one alternative is better than 


the other one.®8 


There are two phases in every outranking method: 


e “The construction of an outranking relation, 
° The exploitation of this relation in order to assist the decision- 
maker.”6? 


The most common outranking methods are the preference ranking 
organization method for enrichment evaluation (PROMETHEE) and the elimination and 


choice translating reality (ELECTRE) method.”” 


Both methods involve a set of alternatives and decision criteria that are 
used as basis for the evaluations. These criteria can be descriptive or ordinal and may 
involve uncertainty. The uncertainty in the criteria is handled with a threshold model that 


involves indifference and preference thresholds. An indifference threshold is the point 





67 “Multi-criterion decision-making using ELECTRE,” University of Geneva, http://ecolu- 
info.unige.ch/~haurie/mutate/Mutate_final/Lectures/Lect_1 3 2/lect_1_ 3 2.htm (accessed December 28, 
2007). 





68 Annika Kangas, Jyrki Kangas, and Jouni Pykalainen, “Outranking Methods As Tools in Strategic 
Natural Resources Planning,” Silva Fennica 35, no. 2 (2001); 216, 
http://www.metla.fi/silvafennica/full/sf35/sf352215.pdf (accessed December 29, 2007). 


69 J. P. Brans and Ph. Vincke, “A Preference Ranking Organization Method: (The PROMETHEE 
Method for Multiple Criteria Decision-Making),” Management Science 31, no. 6 (1985);. 648, 
http://links. jstor.org/sici?sici=0025-1909(198506)31%3A6%3C647%3 AAPROM(%3E2.0.CO%3B2-B 
(accessed December 29, 2007). 








70 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 365. 
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below which the two alternatives are indifferent to the decision maker and the preference 
threshold is the point above which an alternative is dominant and preferable over another 
alternative. Between these two thresholds there is a zone in which the decision maker can 
not decide and it is called the zone of weak preference. However, in the case where the 


criteria are ordinal or descriptive, this zone is not taken into consideration.’! 


(1) Preference Ranking Organization Method for Enrichment 
Evaluation (PROMETHEE). This method needs a preference function for each criterion 
to compute the degree of preference.’? “The credibility of the outranking relation that 
alternative a, is better than alternative a; is described by the outranking degree and it is 


calculated as; 


Pp 
[1 (4-4) =>) WF; (a,.4)) 
j=l 


where Fj(ax, ai) 1s the preference function and w; is the relative importance of the different 


criteria.’”’73 


The value of preference functions is calculated using thresholds p; 


and q; as 74 


8 ;(a) 


P,-4; 





, otherwise 


71 Kangas, Kangas, and Pykalainen, “Outranking Methods,” 217. 


72 Jean-Pierre Brans and Bertrand Mareschal, “How to Decide with PROMETHEE,” 
VisualDecision.com. 2, http://www.visualdecision.com/Pdf/How%20to%20use%20PROMETHEE. pdf 
(accessed December 26, 2007). 


73 Kangas, Kangas, and Pykalainen, “Outranking Methods,” 218. 
74 Tid. 
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The outranking degree that is calculated using this value of 
preference function and relative importance is then used to calculate the positive, 


negative, and net preference flows as shown below:’> 


The positive flow shows how much an alternative is more powerful 
than the other alternatives while the negative flow shows how much weaker it is.’ An 
alternative outranks the other one if its net flow is higher than that of the other one. On 
the other hand, if their net flows are equal, they are indifferent to the decision maker. 
Therefore, the alternative that has the highest net flow is considered to be the best 


choice. 77 


(2) The Elimination and Choice Translating Reality (ELECTRE). 
When decision criteria are either quantitative or qualitative and not even comparable, this 
outranking method can be used to order all alternatives.’8 In the ELECTRE method, the 
concordance and discordance indices as well as the threshold values are used to draw 
graphs for strong and weak relationships. After that, alternatives are ordered in a repeated 


process with the assistance of these graphs.’? 


An index of concordance that demonstrates that an alternative is at 
least as good as another one is calculated for each pair of alternatives. The formula used 


to compute the concordance index is shown below: 


P 
C(a,,4,) = >o wc; (a.4,) 
j=l 


75 Kangas, Kangas, and Pykalainen, “Outranking Methods.” 

76 Brans and Mareschal, “How to Decide with PROMETHEE,” 2. 

77 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 371. 
78 University of Geneva, “Multi-criterion decision-making using ELECTRE.” 

79 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 371. 
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where cj(ax,a1) is the local concordance index and w; is the relative importance of the 


different criteria.8° 


The value of the local concordance index is calculated using 


thresholds p; and qj as®! 


Lifg,(4,)—8;(4,)2 p, 
F,(a,,4,) =40,ifg (a,)-—8; (a) <4, 
8 (4,)-8;(4,)—-4 

P,-4, 





! otherwise 


As a result, binary outranking relations between the alternatives are 


provided with this method.*2 


d. The Technique for Order Preference by Similarity to Ideal 
Solutions (TOPSIS) 


TOPSIS was developed as an alternative to ELECTRE by Kwangsun 
Yoon and Hwang Ching-Lai in 1980. The basic concept of this method is that the most 
preferred alternative should not only have the shortest distance from the positive ideal 
solution (the closest to positive ideal), but also have the longest distance from the 
negative ideal solution (the farthest to negative ideal). This technique has been widely 
used in various multi-criteria decision making models due to its simplicity and 
comprehensibility in concept, its computational efficiency, and its ability to measure the 


relative performance of the decision alternatives in a simple mathematical form.®3 


80 Kangas, Kangas, and Pykalainen, “Outranking Methods,”219. 
81 Kangas, Kangas, and Pykalainen, “Outranking Methods.” 
82 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 372. 


83 Yu-Hern Chang and Chung-Hsing Yeh, “Evaluating Airline Competitiveness Using Multi-Attribute 
Decision Making,” Omega the International Journal of Management Science 29 (2001); 409, 
http://www.elsevier.com/locate/dsw (accessed January 4, 2008). 


a7 





The assumption of the method is that each attribute has a monotonically 
increasing or decreasing utility which makes it easy to locate the positive and negative 
ideal solutions. The preference order of alternatives is found by comparing the Euclidean 
distances of alternatives to the ideal solutions (positive and negative). A decision matrix 
of M alternatives and N criteria is formulated firstly. Then the decision matrix is 
normalized and the weighted decision matrix is constructed. Next, the ideal and negative 
ideal solutions are found. The decision maker seeks to have maximum values for the 
benefit criteria and minimum values for the cost criteria among the alternatives. Then, 
separation measures are calculated, and finally relative closeness of each alternative to 
the ideal solution is determined. The best alternative (the alternative with the highest 
rank) is the one which has the shortest distance to the ideal solution and the longest 


distance to negative ideal solution.*4 
e. Compromise Programming (CP) 


Compromise Programming (CP) is a multiple criteria decision making 
(MCDM) approach introduced by Yu and Zeleny in the 1970s. The main idea behind this 
approach is to determine a subset of efficient solutions (compromise set) that are nearest 
to an ideal and infeasible point (ideal point) in which all the criteria are optimized.®5 It 
defines the best solution as the one whose point is the least farthest from the ideal point in 
the set of efficient solutions. Finding a solution that is as close as possible to ideal is the 
aim of the approach. The distance measure used in CP is the family of L,-metrics 


formulated below:%° 
i 
L, (a) a, If, —f(a)\/|M, —m,| 
J= 


where L, (a) is the L, metric of alternative a, f(a) is the value of alternative a for criterion 


J, Mj; and m; are the maximum (ideal) value and the minimum (anti ideal) value of 


84 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 372. 


85 Francisco J. André and Carlos Romero, “On the Equivalence between Compromise Programming 
and the Use of Composite Compromise Metrics,” (working paper, series: WP ECON 06.33, 2006), 2 
http://ideas.repec.org/s/pab/wpaper.html (accessed December 25, 2007). 





86 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 372. 
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criterion j in set A, f;* is the ideal value of criterion j, w; is the weight of the criterion /, 
and p is the parameter reflecting the attitude of the decision maker with respect to 
compensation between deviations. For p = /, all deviations from f;* are taken into 


account in direct proportion with their assigned weights.°” 
f Multi-Attribute Utility Theory (MAUT) 


Multi-attribute utility theory is another popular method for decision- 
making that is used to determine the best alternative among many and takes into 
consideration the decision maker’s preferences in the form of the utility function which is 
defined over a set of attributes. In this method, all single attributes of an alternative (a 


product or a service) are evaluated to reach a final decision.88 


It involves the following set of methods to make the best decision: 


e “Define the alternatives and relevant attributes. 

e Evaluate each alternative on each attribute. Remove dominated 
alternatives. 

e Assign relative weights to the attributes. 

e Combine the attribute weights and evaluations to yield an overall 


evaluation of each alternative. 


e Perform sensitivity analysis and make a decision.”8? 


The final value of an alternative is calculated by using the following 


formula:29 





87 Pohekar and Ramachandran, “Application of Multi-Criteria Decision Making,” 372. 


88 Bobbi Blaser Johnson, Eli Gratz, Kathy Longenecker Rust, and Roger Smith, “Multiattribute Utility 
Theory,” (technical report, University of Wisconsin), 
http://www.12d2.uwm.edu/atoms/archive/technicalreports/fieldscans/tr-mau.html#maut_at_outcomes 
(accessed December 11, 2007). 


89 «Multi-Attribute Utility Theory,” University of Illinois, 
http://pennmush.tinymush.org/~alansz/courses/mdm-block/maut.html (accessed December 11, 2007). 








90 Ralph Schafer, “Rules for Using Multi-Attribute Utility Theory for Estimating a User’s Interests,” 
Distributed System Institute, 3, http://www.kbs.uni- 
hannover.de/~henze/ABIS_Workshop2001/final/Schaefer_final.pdf (accessed December 16, 2007). 
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where v(x) is the value (utility) function of an alternative against the i” value dimension 
and w; is the weight assigned to that value dimension. The sum of the all weights 
assigned to each value dimension equals to one and v(x) is calculated by using relevant 
attributes of the value dimensions as?! 


(x)= 2) WaPai(7(@)) 


aceA; 


Here, A; is the set of all attributes relevant for dj, vg; (I(a)) is the evaluation 
of the actual level /(a) of attribute a on d;, and wa; is the weight that determines the 


impact of the evaluation of attribute a on value dimension dj. The variable wz; is also 


called the relative importance of attribute a for dj. For all d; (i=J,...,n) holds De w,, =1. 
ac, 


3. An Alternative Approach to Multi-Criteria Decision Making 
Techniques: Decision Support Systems (DSS) and Expert Systems 


In some cases, decision makers do not have the adequate background knowledge 
to be able to understand and use the techniques which are briefly summarized above. 
Instead, these decision makers can use some software tools, which are built to use the 
techniques (one or some of them) with a user friendly interface, to get the support to 


make a decision. These support tools are decision support systems and expert systems. 


“Decision Support Systems (DSS) are interactive information systems that rely on 
an integrated set of user-friendly decision support tools to produce and present 


information to support management in the decision making process.” 


Most of the time, managers rely on their personal experience and the information 
that is available to help them in making decisions, but sometimes when encountering a 


real complex decision, their human abilities are not capable of evaluating for the best.?3 





91 Schafer, “Rules for Using Multi-Attribute Utility Theory,” 3. 


92 Larry Long and Nancy Long, Computers Information Technology in Perspective (New Jersey: 
Pearson Education Inc. Press, 2005), 375. 


93 Tid. 
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When it is a complex decision, DSS helps decision makers to choose between 
alternatives and can even rank alternatives based on the user’s preferences. Also, DSS 
can help managers to close the gap between the necessary information and available 


information.?4 


Decision support systems generally consist of a database, a knowledge base, a 
model base, and a user interface. The database provides the relevant historical data for the 
system to use in the model base. The knowledge base keeps the rules that are constraining 
the solution alternatives. The model base includes statistical, forecasting, simulation, and 
other math tools. The model base also provides users with appropriate tools without 
developing a model from the beginning. The user interface allows users to control the 


DSS easily and help them to decide which data and techniques to use in the analysis. 


The most important characteristics of DSS are helping decision makers, especially 
at tactical and strategic levels, in addressing semi-structured and unstructured problems, 
being interactive and user friendly, and being adaptable to meet the requirements of any 


decision.2© 


Data management, modeling, statistical analysis, planning, inquiry, representing 
information, and consolidation of similar information are the most important and 


common capabilities of DSS.97 


Another type of information systems is Expert Systems (ES). “An Expert System 
is an interactive system that responds to questions, asks for clarification, makes 
recommendations, and generally helps the users in decision making process.”?8 Expert 


systems are the simulation of the human thinking process, and it is like working with a 


94 Long and Long, Computers Information, 375. 
95 Spainhower, “An Exploratory Study,” 42. 

96 Long and Long, Computers Information, 375. 
97 Tbid., 376. 

%8 Tbid., 379. 
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human expert to solve a problem. But, expert systems are better because they do not miss 
any important consideration or alternative. Expert systems are using if-then step-by-step 


rules to solve the problems.?? 
C. CONCLUSION 


Multi-criteria decision analysis (MCDA) offers numerous methodologies and 
schools of thought for multi-criteria modeling and decision support, as the authors 
attempted to discuss the most popular ones in this chapter. The approaches vary from 


very simple ones to very complex optimization models. 


The main aim of this study is to show the subjectivity involved in offeror 
(proposal) evaluation in the source selection part of government acquisitions. Therefore, 
the authors preferred to choose a technique that is simple to understand by anyone who 
does not have a multi-criteria decision making (MCDM) background. WSM is one of the 
very basic models/techniques used in situations where more than one criterion is involved 
and the relative importance of each criterion is different from each other. The model’s 


biggest strength is its simplicity in application. 


In the last thirty years, very different methods were developed in the area of 
MCDM. Most of the improvements are related with the setting of priorities (relative 
importance). Whenever the number of criteria increases and subcriteria exists, providing 
consistency of the relative importance becomes harder and harder. As stated before, this 
technique is one of the very basic methods; it is not providing a decision maker with a 
tool to determine the weights consistently. The decision maker has the responsibility to 
provide the consistency which is especially needed while assigning weights that is 


supposed to reflect their relative importance. 


99 Long and Long, Computers Information. 
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In this chapter, an overview on the available multi-criteria decision support tools 
and techniques was given. Among all these techniques, in this study one of the basic 
methods will be used to help the authors show the aim of their study. The next chapter 
will discuss the Air Force’s KC-X program in which the authors will use a simplified 


version of it in their study as a sample case. 
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IV. USAF’S KC-X NEXT GENERATION TANKER PLANES 
PROGRAM 


A. INTRODUCTION 


In this part of the study, the authors will give some background information about 
the Air Force’s KC-X program. The authors will use this program in their model to show 
the subjectivity involved in the source selection proposal evaluation part of Air Force’s 
acquisitions. First, some introductory information about the program will be given, and 
then, the requirements will be explained in moderate detail. Next, background on how the 
competition began between two offerors will be discussed, and lastly, some specific 
information (to roughly compare the two alternatives) about each proposed aircraft will 


be presented. 


The aim of the authors’ project is to show the effects of the subjectivity involved 
in government procurement processes. Subjectivity involvement in the evaluation of the 
proposals in the source selection part of an acquisition is inevitable. However, 
subjectivity contrasts with fairness and best value. The authors try to show the effects of 
subjectivity involved in the process of selection by using a simplified and current real-life 


program. 


The authors chose the USAF’s KC-X program to study based upon their research. 
There are some specific reasons for the authors to choose that program. First of all, they 
wanted to show the effects of subjectivity, so they decided to choose an Air Force 
procurement program in which color ratings were used. This is because in the authors’ 
perspective the Air Force’s color rating is one of the most subjective techniques used in 


the government source selection. 


The authors decided to study a program that was popular and current, as it was 
thought that they could have lots of different comments from a multitude of viewpoints. 
With a current program, they hoped to have more insight information, which was 
publicly available, about the program. Otherwise, most of the information about old 


programs is not accessible online because of confidentiality issues. Also, it was very clear 
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that they would not be able to get any official information about an ongoing program but 
they thought that they could get enough data to do this study from the publicly available 
part of the data about the program. After all these considerations, they came up with the 


idea of doing their study on USAF’s KC-X program. 


After doing some research about the program, they realized that the two 
alternatives offered very different specifications and capabilities for the same program. 
This showed the authors the difference of the two offeror’s point of view. The authors 
thought that this program might be a good sample for their study because of the potential 
effects of subjectivity. 


Another factor that made the authors’ choice even more interesting for the study 
is the high potential of the program to get protested. Experts of the defense industry were 
expecting a protest regardless of the winner. Since the proposed KC-X alternatives have 
very different strengths in terms of capabilities, results would not easily satisfy the 


offeror who lost the program award. 


The acquisition of weapon systems has always been a crucial part of government 
acquisitions. They are crucial not only because of the dollar amount that goes along with 
these acquisitions, but also because the final product will be an important determinant 
factor in national security. Also, these programs take too much time and effort. 
Consequently, these projects may keep the U.S. as a super power and pioneer new 
technologies when they have successfully ended, but may keep the U.S. behind when 
they are not successful. There are other important factors that have impact on these 
processes like not having enough independent offerors and competition. Therefore, these 


projects are necessary but risky. 


The Department of Defense (DOD) and the Air Force have become increasingly 
concerned about their aging aerial refueling aircraft because of the heavy pace of 


operations in support of the war on terrorism. The aerial refueling fleet was last reviewed 
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by the Government Accountability Office (GAO) in 1996 and it was found that KC-135 


aircraft were aging and becoming increasingly costly to maintain and operate.!00 


Currently, one of the biggest weapon systems projects is the U.S. Air Force’s KC- 
X program. In January 2007, the solicitation for the KC-X project was advertised on the 
Federal Business Opportunities (FBO) website. The program has been planned for the 
production of 175 airplanes and 4 test platforms (i.e., a total of 179 airplanes). The cost 
objective for the first phase was approximately $40 billion. When the program was 
announced, the USAF declared their thoughts about renewing the Air Force’s tanker fleet 
(America’s current aerial tanker fleet is approximately 40-50 year old) project as their 


number one priority.!0! 


The releases stated that “the Air Force also intends to take full advantage of the 
other capabilities inherent in the platform, and make it an integral part of the Defense 
Transportation System.” Contending aircrafts were offering substantial improvements 


over the KC-135’s extra capacity for cargo or people, in addition to their tanker roles.!02 


Lt. Gen. Donald Hoffman, the Assistant Secretary of the Air Force for 
Acquisition, announced the posting of the KC-X Aerial Refueling Aircraft RFP to the 
FBO website on January 30, 2007, signaling the official launch of the Air Force’s number 
one priority acquisition program. The release noted that “The KC-X program is the first 
of three acquisition programs the Air Force will need to replace the entire fleet of aging 
KC-135 Stratotankers, which have been in service for more than 50 years”. The RFP 


stipulated nine primary key performance parameters:!% 


1. Air refueling capability 
2. Fuel offload and range at least as great as the KC-135 





100 U.S. General Accounting Office (GAO), MILITARY AIRCRAFT: DOD Needs to Determine Its 
Aerial Refueling Aircraft Requirements, GAO-04-349 Report June 2004, 
http://www.gao.gov/new.items/d04349.pdf (accessed March 11, 2008). 


101 August Cole, “Airbus to Raise Ante in Bid for Military-Tanker Deal”, Wall Street Journal, January 
14, 2008, 6, http://ebird.afis.mil/ebfiles/e20080 114572969. html. (accessed January 21, 2008). 


102 “The USAF’s KC-X Aerial Tanker RFP,” http://www.defenseindustrydaily.com/the-usafs-kex- 
aerial-tanker-rfp-03009/ (accessed February 1, 2008). 


103 “The USAF’s KC-X Aerial Tanker RFP.” 
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3. Compliant communication, navigation, surveillance/air _ traffic 
management (CNS/ATM) equipment 


Airlift capability 
Ability to take on fuel while airborne 
Sufficient force protection measures 


Ability to network into the information available in the battlespace 


ou ON. 


Survivability measures (defensive systems, Electro-Magnetic Pulse (EMP) 
hardening, chemical/biological protection, etc,) 


Zs Provisioning for a multi-point refueling system to support Navy and Allied 
aircraft 


The USAF said that final RFP defined an integrated, capability-based, best-value 
approach, and included specific factors for assessing the capability contribution of each 


offeror, along with cost and assessments of past performance and proposal risk. 


Also, it was announced by the Air Force that the department had gone through a 
careful review process for KC-X and had validated that the RFP had accurately reflected 


the requirements as laid out by the warfighter. 


The Boeing 707-based KC-135 fleet ranges in age from 40 to 50 years old, raising 
the risk that fatigue or aging-related problems could ground them at some unanticipated 
time. Since aerial transport, fighter strike missions, bomber missions, and combat air 
patrols all depend on aerial refueling to some degree, a grounding of the KC-135 fleet 


could be catastrophic for America’s military posture.!% 


The primary mission of the KC-X will be to provide aerial refueling to the United 
States military and coalition aircrafts in the war on terrorism and other missions. 
However, the Air Force also intends to take full advantage of the other capabilities 
inherent in the platform and make it an integral part of the Defense Transportation 
System. The RFP identified nine primary key performance parameters: addressing air 


refueling capabilities, including fuel, receiving fuel by aerial refueling from other 





104 “USAF KC-X: Will There Be a Competition?” http://www.defenseindustrydaily.com/usaf-kex- 
will-there-be-a-competition-02994/ (accessed February 1, 2008). 
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platforms, and range at least equal to those of the existing KC-135. A new capability 
addressing joint operations will be the provisioning for a multi-point refueling system, to 


support Navy and allied aircraft. ! 
B. USAF’S REQUIREMENTS FOR KC-X 


According to the Headquarters Air Mobility Command White Paper on KC-X, 
Joint Doctrine is the root of the requirement for a flexible Aerial Refueling Aircraft that 
can operate throughout a battlespace to deliver fuel and/or cargo and/or passengers. The 
Nation’s new KC-X aircraft, which is required to be equipped with appropriate floors for 
carrying passengers and cargo, reasonably-sized doors to accommodate standard-sized 
pallets, and modest defensive systems which allow the aircraft access to an area of 


operations, will help US Combatant Commanders to success in their mission. ! 


Also, this document states that, US Joint Publications recognized the need for a 
refueling tanker to be able to manage different types of missions at the same time and 
have transferred these valuable lessons into current doctrine. For example, JP 3-17, one 
of the joint doctrines related with air mobility, states that all USAF tanker aircrafts are 
capable of performing an airlift role and are used to augment core airlift assets. Dual role 
concept demands an air refueling aircraft that can transport a combination of passengers 
and cargo while performing air refueling. The same concept states that in some 


circumstances, it may be more efficient employing an air refueling aircraft strictly in an 


airlift role. Air refueling units’ deployment because of their organic capacity may be 
more effective to transport unit personnel and support equipment or passengers and cargo 


from other units.!°7 


105s Air Force's KC-X Aerial Tanker Replacement Program.” http://www.defense- 
update.com/newscast/0107/news/300107_kcx.htm (accessed February 1, 2008). 


106 Headquarters Air Mobility Command White Paper, “KC-X: The Next Mobility Platform the Need 
for a Flexible Tanker.” http://www-tc.pbs.org/newshour/bb/military/tanker_kc-x.pdf (accessed March 11, 
2008). 


107 Headquarters Air Mobility Command White Paper, “KC-X: The Next Mobility Platform the Need 
for a Flexible Tanker.” 
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The USAF announced its final solicitation (RFP) on January 30, 2007, and in its 
RFP, the USAF explained the requirements for the KC-X. Mission capability, proposal 
risk, past performance, cost/price, and the integrated fleet aerial refueling assessment 
(IFARA) will be the five main factors for the USAF’s best value product evaluation for 
this program. The USAF explained this in their RFP numbered FA8625-07-R-6470 and 
dated January 30, 2007, as stated below, 


The Government will select the best overall offer, based upon an integrated 
assessment of Mission Capability, Proposal Risk, Past Performance, 
Cost/Price and the Integrated Fleet Aerial Refueling Assessment (IFARA). 
Contract(s) may be awarded to the offeror who is deemed responsible in 
accordance with the FAR, as supplemented, whose proposal conforms to the 
solicitation's requirements (to include all stated terms, conditions, 
representations, certifications, and all other information required by Section L 
of the solicitation) and is judged, based on the evaluation factors and 
subfactors, to represent the best value to the Government. The Government 
seeks to award to the offeror who gives Air Force the greatest confidence that 
it will best meet, or exceed, the requirements. This may result in an award to 
a higher rated, higher priced offeror, where the decision is consistent with the 
evaluation factors and the Source Selection Authority (SSA) reasonably 
determines that the technical superiority and/or overall business approach 
and/or superior past performance, and/or the IFARA of the higher priced 
offeror outweighs the cost difference. The SSA will base the source selection 
decision on an integrated assessment of proposals against all source selection 
criteria in the solicitation (described below). While the Government source 
selection evaluation team and the SSA will strive for maximum objectivity, 
the source selection process, by its nature, is subjective and, therefore, 


professional judgment is implicit throughout the entire process. 108 


The USAF will make the award to the best valued option. The evaluation will be 
made as explained in the RFP Section M. The main factors are given in the order of 
importance (mission capability, proposal risk, past performance, cost/price and IFARA) 
and for the USAF, the mission capability, proposal risk, and past performance evaluation 
factors are of equal importance and the cost/price and IFARA evaluation factors are also 
of equal importance. The first three evaluation factors are more important than the other 


two factors. The USAF explained this in their RFP as stated below, 


108 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 1. 
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Award will be made to the offeror submitting the most advantageous 
proposal to the Government based upon an integrated assessment of the 
evaluation factors and subfactors described below. The Mission 
Capability, Proposal Risk, and Past Performance evaluation factors are of 
equal importance and individually more important than either Cost/Price 
or IFARA evaluation factors individually. The IFARA is equal in 
importance to Cost/Price. Within the Mission Capability factor, the five 
(5) subfactors are listed in descending order of relative importance from 1 
to 5. In accordance with FAR 15.304(e), the Mission Capability, Proposal 
Risk, Past Performance, and IFARA evaluation factors, when combined, 
are significantly more important than Cost/Price; however, Cost/Price will 
contribute substantially to the selection decision.!9 


Each of the five factors will be discussed separately to give more specific 


information about each of them and how they should be evaluated. 
1. Factor 1: Mission Capability 


The offeror's capability to satisfy the government's requirements will be assessed 
by the mission capability evaluation. This factor has 5 subfactors to evaluate. All mission 
capability subfactors will each receive one of the color ratings described in Air Force 
Federal Acquisition Regulation Supplement (AFFARS) in Mandatory Procedures part 
MP5315.3, paragraph 5.5.1 (blue-exceptional, green-acceptable, yellow-marginal, red- 
unacceptable). “Assessment will focus on the strengths and deficiencies of the offeror's 
proposal. The color rating represents how well the offeror's proposal meets the Mission 


Capability subfactor requirements.” Mission Capability subfactor 5 will only receive 1 of 


the following 3 ratings: acceptable (green), marginal (yellow), or unacceptable (red)). 
There will not be any overall color rating, which is rolled up from subfactor ratings, for 


the mission capability factor.!!9 
a. Subfactor 1: Key System Requirements 


The first subfactor under mission capability is key system requirements. 


The offeror’s understanding of the requirements (as defined in the system requirements 


109 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 2. 
110 Thid., 3. 
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document (SRD)) and how its ability to meet those requirements has substantiated will be 
evaluated under this subfactor. Only the logistics requirements as addressed in the 


Product Support section of subfactor 3 will not be evaluated herein subfactor 1.!!! 


The government’s evaluation of the offeror's approach to meet SRD 
requirements, according to the USAF’s solicitation (RFP) document, numbered FA8625- 
07-R-6470, explained below. 


Aerial refueling will include tanker aerial refueling, receiver aerial 
refueling, fuel offload versus radius range, drogue refueling systems (including 
simultaneous multipoint refueling), the operationally effective size of the boom envelope, 
the aerial refueling operator station and aircraft fuel efficiency. Airlift will include airlift 
efficiency, cargo, passengers, aero-medical evacuation, ground turn time, and cargo bay 
re-configuration. Operational Utility will include aircraft maneuverability, worldwide 
airspace operations, communication/information systems (including Net-Ready 
capability), treaty compliance support, formation flight, intercontinental range, 7,000 foot 
runway operations, bare base airfield operations, and growth provisions for upgrades. 
Survivability will include situational awareness, defensive systems against threats, 
chemical/biological capability, EMP protection, fuel tank fire/explosion protection, and 
night vision capability. Other system requirements will include all other SRD 
requirements that are not above (in Aerial Refueling, Airlift, Operational Utility and 


Survivability) or in Subfactor 3.!!2 
b. Subfactor 2: System Integration and Software 


The second subfactor under mission capability is system integration and 
software. This subfactor will evaluate the offeror’s ability to implement disciplined and 
institutionalized systems engineering approach and the offeror's capability to manage and 
integrate the software elements to satisfy performance capability requirements in the KC- 


X’s SRD. The USAF explained this in their RFP as stated below.!!3 





111 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 3. 
112 Thid., 6. 
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The Government will evaluate the proposal to determine the offeror's 
ability to implement a disciplined and _institutionalized systems 
engineering approach necessary to successfully design, develop, integrate, 
validate and verify requirements, manufacture, and sustain the KC-X 
system as defined by the performance capability requirements set forth in 
the KC-X SRD. The software development capability (SDC) will be 
evaluated to determine the offeror's capability to manage and integrate the 
software elements required to satisfy the performance requirements. !!4 


c. Subfactor 3: Product Support 


Another subfactor under mission capability is product support. The 
offeror's proposed product support approach will be evaluated for an efficient, effective, 
and comprehensive support program for the service life of the KC-X fleet. The offeror's 
approach to achieve reliability, availability, maintainability, and supportability with an 
optimal logistics footprint will be evaluated by the government under this subfactor. 
Operational availability (Ao), reliability and maintainability (R&M), and mission 


capability (MC) rate will be the determination points of the government’s evaluation.!!5 
d. Subfactor 4: Program Management 


The program management approach that the offeror will use for the KC-X 


program is also another important part of the mission capability factor. It is the fourth 


subfactor of mission capability. The government will evaluate the offeror's proposal to 
determine if the offeror is using a realistic and reasonable approach to effectively and 


efficiently implement and manage the KC-X program.!!6 


The USAF explained how the realism and reasonableness determinations 


will be done in their RFP as stated below. 


Realism will be assessed to ensure the offeror's proposal reflects a clear 
understanding of program requirements, correlates to other program 
documentation, and is consistent with the approach described in the 


114 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 5. 
115 Tbid., 6. 
116 Thid., 7. 
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technical volumes. Reasonableness will be assessed to ensure the 
proposed logic and methodology reflected in program documentation is 
acceptable and reflects an understanding of commonly accepted program 
management concepts and practices (emphasis added).!!7 


e. Subfactor 5: Technology Maturity and Demonstration 


The last of the subfactors of mission capability is technology maturity and 
demonstration, and this subfactor will evaluate the maturity of the critical technology 
elements (CTE) that are included in the offeror's proposed KC-X aircraft. These critical 
technology elements are described in the DoD Technology Readiness Assessment (TRA) 
Deskbook that can be accessed from the DoD’s website. A CTE is an element that is new 
or novel or is being used in a new or novel way. It is critical if it is necessary to achieve 
the successful development of a system, its acquisition, or its operational utility. For 
example, a few of these can be mechanical components, processors, servers, electronics, 
and software (algorithm) interfaces. But, with the guidance provided by the DoD 
Technology Readiness Assessment (TRA) Deskbook, these elements are assessed for 


each program specifically. !!8 
2. Factor 2: Proposal Risk 


A proposal risk factor evaluation will be done at the mission capability subfactor 
level for only subfactors 1-4. The weaknesses associated with an offeror's proposed 
approach will be focused on while doing the proposal risk evaluation. This evaluation 
will try to determine the risks involved in program management objectives like cost, 
scheduling, and performance. An assessment of the potential for disruption of schedule, 
increased cost, degradation of performance, and the need for increased government 
oversight, as well as the likelihood of unsuccessful contract performance, will be the key 


points of this evaluation.!!9 


117 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 7. 
118 Thid., 9. 
119 Thid., 9. 
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The first four mission capability subfactors will receive one of the proposal risk 
ratings (high, moderate, low) described in AFFARS MP5315.3, paragraph 5.5.2. The 
risks related with the weaknesses or significant weaknesses of the offeror's proposed 
approach to each of the mission capability subfactors will be the focus of the evaluation. 
If any weakness is identified, then the evaluation will also focus on the offeror's proposed 


improvements and whether that approach is manageable or not. 
3. Factor 3: Past Performance 


The degree of confidence the government has in an offeror's ability to provide the 
product or service that meets warfighters' needs, including cost and schedule, is 
determined by the past performance evaluation factor and based on records of 


demonstrated performance. !20 


The Performance Confidence Assessment Group (PCAG) is the government 
evaluation team that will conduct an in-depth review and evaluation of all obtained 
performance data. The PCAG will work on both past and present performance data 
identified by offerors in their proposals and additional past and present performance data, 
if available from other sources. If necessary, the PCAG will also confirm the past and 
present performance data identified by offerors in their proposals. Additionally, the 
offeror's performance in managing and mitigating program risk will be assessed. After an 
evaluation of the offeror's recent past performance relevant to only mission capability 
subfactors 1-4 and cost/price, the government will assess an overall performance 
confidence. The overall past performance evaluation results will be the government's 
confidence in the offeror's ability to fulfill the solicitation requirements while meeting 


schedule, budget, and performance quality constraints.!2! 


To also understand the government’s evaluation in a better way, some notions like 


performance confidence assessment, relevance, and offerors without a record of relevant 


120 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 9. 
121 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 10. 


a) 


past performance and performance problems should be understood correctly. The AF 


explained these in their RFP as stated below, 


Performance Confidence Assessment - under the past performance 
factor, the performance confidence assessment represents the evaluation of 
an offeror's present and past work record in order to assess the 
government's confidence in the offeror's probability of successfully 
performing as proposed (emphasis added). !22 


The performance confidence assessment will be assessed at the overall 
factor level after evaluating aspects of the offeror's recent past 
performance, focusing on performance that is relevant to the mission 
capability subfactors 1-4 only and cost or price.... Each offeror will 
receive one of the confidence ratings (high confidence, significant 
confidence, satisfactory confidence, unknown confidence, little 
confidence, or no confidence) as prescribed in AFFARS MP 5315.305 
(August 10, 2005).!23 


Relevance - the past performance evaluation is accomplished by 
reviewing aspects of an offeror's and major/critical subcontractor's 
relevant present and recent past performances (emphasis added).!24 


Offerors without a record of relevant past performance or for whom 
information on past performance is not available will not be evaluated 
favorably or unfavorably based on past performance and, as a result, will 
receive an "unknown confidence" rating for the past performance factor 
(emphasis added).!25 


Where relevant performance records indicate performance problems, the 
government will consider the number and severity of the problems as well 
as the appropriateness and effectiveness of any corrective actions taken 
(not just planned or promised). The government may review more recent 
contracts or performance evaluations to ensure corrective actions have 
been implemented and evaluate their effectiveness (emphasis added).!26 


122 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 10. 
123 hid. 
124 hid, 
125 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 11. 
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4. Factor 4: Cost/Price 


The cost/price evaluation will be done by using criteria to evaluate the current 
program costs, then the most probable life cycle cost (MPLCC) evaluation will be done, 
and finally proposals will be analyzed to determine whether or not they are unbalanced 


with respect to prices, quantity matrix factors, or separately priced line items. 


Criteria, MPLCC and unbalanced pricing are the important points of the cost/price 
evaluation and the USAF explained these in their RFP as stated below. 


e Criteria - the cost panel will evaluate the offeror's cost proposal against 
realism, reasonableness, and cost/price risk rating.!27 


e Most Probable Life Cycle Cost (MPLCC) is an independent government 
cost estimate, which is adjusted for technical, cost, and schedule risks, to 
include all contract, budgetary, and other government costs. Costs those 
are associated with all phases of the entire weapon system life cycle will 
be included. A work breakdown structured approach will be used to come 
up with an independent government cost estimate. !28 


@ Unbalanced Pricing - proposals will be analyzed to determine whether or 
not they are unbalanced with respect to prices, quantity matrix factors, or 
separately priced line items. Unbalanced pricing exists whenever the total 
solicitation effort price is acceptable but, the price of one or more contract 
line items or a factor in the quantity matrices is significantly overstated or 
understated (can be demonstrated by the application of price analysis 
techniques in FAR 15.404-1 (b)).!29 


5. Factor 5: Integrated Fleet Aerial Refueling Assessment 


The integrated fleet aerial refueling assessment (IFARA) factor evaluation will be 
done by using a scenario-based simulation program. An offeror’s KC-X fleet data from 
its proposal will be evaluated on a specific scenario by this simulation program to 
determine the number of KC-X to meet peak demand for mission accomplishment. The 
same scenario will evaluate the number of KC-135R needed to accomplish the mission 


instead of KC-X. For comparison, the number of KC-135R needed for the mission will be 


127USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 12. 
128 Thid., 13. 
129 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 14. 
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divided by the number of KC-X needed. This ratio will be called the fleet effectiveness 
value and will show the effectiveness of KC-X instead of KC-135R. A fleet effectiveness 
value of 1.0 will be assessed as equal in effectiveness to the KC-135R, so a fleet 
effectiveness value greater than 1.0 will be assessed as more effective than the KC-135R 
and more advantageous to the government. The USAF explained the IFARA factor in 
their RFP as stated below, 


The Government will use modeling and simulation to provide an 
integrated assessment of the utility and flexibility for a fleet of the 
offeror's proposed KC-X by evaluating the number of aircraft required to 
fulfill the peak demand of the aerial refueling elements evaluated in the 
2005 Mobility Capability Study (MCS). In the context of this evaluation 
scenario, the Government will determine the proposed KC-X's fleet 
effectiveness in relation to a KC-135R fleet. !30 


The Government will conduct the analysis using offeror-provided data in 
the evaluation scenario by primarily using the Combined Mating and 
Ranging Planning System (CMARPS) modeling and simulation tool. The 
results of the CMARPS evaluation will provide the Government with the 
quantity (based on the offeror-proposed KC-X aircraft) required to meet 
the mission requirements of the evaluation scenario. The same scenario 
will be run on CMARPS using a KC-135R fleet to provide a baseline 
quantity for comparison. The required number of KC-135R aircraft 
generated by the model will be divided by the number of proposed KC-X 
aircraft required to meet the same scenario. This ratio is the "fleet 
effectiveness value" for the proposed KC-X aircraft.!3! 


The Government will report the "fleet effectiveness value" as determined 
by the evaluation as a standalone "value" to the SSA, along with any 
major insights and observations gleaned from the evaluation. This value 
will be determined by Government analysis taking into account the 
offeror's input data and considering any analysis performed by the offeror 
of the same evaluation scenario.!32 


130 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 14. 
131 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 15. 
132 Thid, 
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C. BOEING AND NORTHROP GRUMMAN-EADS TEAM COMPETED FOR 
KC-X PROGRAM 


Shortly after the AF released its RFP, Boeing and Northrop Grumman-European 
Aeronautic Defense and Space Company (EADS) announced that they would bid on the 
program, and the two teams began competing for this contract award. On one side was 
the Boeing team with KC-767 Advanced (767-200 derivative) and on the other was the 
Northrop Grumman-EADS’s team with KC-30B (Airbus A330-200/200F derivative). 
Approximately a year after solicitation, final proposals were submitted on January 3, 


2008.!33 


Both teams waited hopefully for the results which were publicly announced on 
February 29, 2008. The Northrop Grumman-EADS (Airbus) team was announced as the 
winner of this program and it was a surprise for lots of people who were expecting 


Boeing to win (Airbus is an aircraft manufacturing subsidiary of EADS). 


The announcement, available on DOD’s website about the USAF’s aerial tanker 


program award, stated the contract award as follows; 


Northrop Grumman Corp., of Los Angeles, Calif., is being awarded a cost 
plus incentive/award fee, fixed price incentive, firm fixed price contract 
for the newly-named KC-45. This contract is awarded after full and open 
bidding, and provides for the system design and development of four test 
aircraft for $1.5B. This contract also includes five production options 
targeted for 64 aircraft at $10.6B. At this time no funds have been 
obligated. Contracting activity is the Aeronautical Systems Center, 
Wright-Patterson Air Force Base, Ohio (contract number FA8625-08-C- 
6451).!34 


On March 7, 2008, Air Force and Boeing officials came together for a formal 
debriefing. After a thorough analysis of data presented at this debriefing on the decision, 
Boeing concluded that the Air Force’s efforts to run a fair, open, and transparent 


competition failed due to irregularities that placed Boeing at a competitive disadvantage 


133 “The USAF’s KC-X Aerial Tanker RFP.” 


4 U.S. Department of Defense, “Contract Announcement webpage,” 
http://www.defenselink.mil/contracts/contract.aspx?contractid=3719 (accessed March 1, 2008). 
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throughout this competition. Boeing claimed that because of these irregularities they were 
penalized for offering a commercial-derivative airplane with lower costs and risks and 


greater protection for troops.!35 


On March 11, 2008, Boeing filed a formal protest to the Government 
Accountability Office (GAO) claiming that there were irregularities with the process of 
the competition and the evaluation of the competitors’ bids and cited these irregularities, 
which are not fully available to the public, in their formal protest. The protest was not a 
surprise for defense industry experts because of the huge difference between the offered 
products of the two competing offerors for the same requirements. Thus, experts were 
expecting a protest from the losing party (either Boeing or NG-Airbus). In the protest, 
Boeing’s main point was the size of NG-Airbus’s KC-X since the AF formerly 
announced this program as a replacement program for the KC-135 and also announced 
that there would be another program to replace KC-10s (KC-10 is an aircraft that is 
almost two times larger than the KC-135). The NG-Airbus team proposed an aircraft that 
was even bigger than the KC-10 for the KC-135 replacement program, so Boeing has 
asked the agency to review the USAF’s decision to award a contract to the Northrop 


Grumman and EADS team to replace aerial refueling tankers.!3¢ 


Boeing’s Vice President and Tanker Program Manager, Mark McGraw, said that 
their analysis of the data presented by the Air Force showed that the competition was 
seriously flawed and resulted in the selection of the wrong airplane for the warfighters. 
He also talked about their concerns about the USAF’s evaluation and expressed that they 
were exercising their protesting process. The GAO will review the decision to ensure that 
the process was fair and the results were the best choice for the U.S. warfighters and 


taxpayers. !37 





135 Boeing, “Boeing Protests U.S. Air Force Tanker Contract Award,” 
http://www.boeing.com/news/releases/2008/q1/080311b_nr.html (accessed March 12, 2008). 


136 Boeing, “Boeing Protests U.S. Air Force Tanker Contract Award.” 
137 oid. 
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D. INFORMATION ABOUT KC-X ALTERNATIVES 


The aim of this project is to show the effects of subjectivity involved in the source 
selection phase of acquisition because of the techniques used in proposal evaluations. The 
authors try to show the effects of subjectivity by using the KC-X program, but they do 
not try to make the proposal evaluation for the KC-X program. Even if they wanted to do 
the evaluation for proposals, they would need lots of experts and much time like the Air 
Force did. Since the authors want to work on realistic data, they use the data about the 
program that was publicly available. The authors expect to have more realistic results this 
way. All the information the authors used were from offerors’ websites for this specific 
program and the product cards of the proposed aircrafts. Also, they have another source 
of information, which is the publicly available part of Boeing’s protest file. The authors 


work on the KC-X program with this publicly available information. 


Since the entire program’s goal is to replace the aging KC-135s with their new 
alternative, all mission capability requirements include the KC-135 as the base point. 
Therefore, the authors provide some very basic capabilities of the KC-135 in this study, 
which will be very helpful for one to understand the USAF’s comparisons as well as the 


capabilities of the new alternatives in a better way. 


While the KC-135s can carry up to 6 standard 463L cargo pallets, 53 people, or 
about 18 medical litters, according to the KC-30 team’s (Northrop Grumman-EADS) 
official brochure, their A330-200/200F derivative can carry up to 32 standard 463L cargo 
pallets, up to 226 passengers, or 120 medical litters, or some combination of the above, in 
addition to its full fuel load. Team Boeing’s KC-767 Advanced can carry up to 19 
standard 463L cargo pallets, up to 190 passengers, or 54 medical litters, or some 


combination of the above, in addition to its full fuel load. 


The exact amount of fuel that the US KC-767 Advanced can carry is not public. 
The base figure of around 200,000 pounds is similar to the existing KC-135, but the 
additional body tank option is a wild card. The KC-767’s advanced refueling boom has a 
900 gallon/minute capacity and been tested successfully in live air-to-air refuelings. 


While the KC-30 can carry over 250,000 pounds of capacity, it also carries more fuel 
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than the 767, which is a particular advantage in the Pacific sector with its wide over- 
water expanses. On the other hand, its advanced 1,200 gallon/minute ARBS refueling 
boom has yet to actually transfer fuel to another aircraft in the air. Figure 3 illustrates the 


size comparison of the KC-30, KC-767AT and KC-135. 





Figure 3. Size Comparison of the KC-30, KC-767AT and KC-135 (Source: Boeing’s 
KC-767’s product card). 


The KC-30 is approximately two times larger than the KC-135 and also 
approximately one and a half times larger than the 767, which requires a slightly longer 
runway and costs more to operate on a per-plane basis. Team KC-30 stresses costs and 
efficiency on a per-mission basis, and if larger tanker aircraft with more fuel and cargo 
space mean fewer sorties required, the figures may look more equal once the mission the 


tankers are supporting is complete. !38 


As their relative capacities demonstrate, the KC-767 is a smaller aircraft than the 
KC-30. One positive consequence is that it can take off from slightly shorter runways. 
The USAF requires the ability to take off from an 8,000 foot runway, but would prefer 


7,000 feet as this makes more runways available. Boeing also claims the KC-767 as being 


138 “The USAF’s KC-X Aerial Tanker RFP.” 
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22 to 24 percent cheaper to operate and maintain on a per-plane basis than the KC-30, 


and its base aircraft is cheaper to buy on the civilian market.!39 


Figure 4 below is the evaluation summary table taken from Boeing’s protest file 
(the publicly available part of the protest file). These are the evaluation summary results 


that Boeing claims that the AF came up with at the end of its evaluations on proposals.!49 


Evaluation Summary of Factors 


Factors 
Mission Capability/Proposal Risk 

















Cost Risk Rating (SDD — P&D) 


IFARA 





Figure 4. | Evaluation Summary Table taken from Boeing’s Protest File. 


This figure was removed from Boeing’s publicly available protest file after the 
USAF and NG separately made objections against it because it included proprietary 


information. 


However, the authors will use these evaluation results from the evaluation 
summary table presented above. In this study, the aim is to show the subjectivity involved 
in source selection, especially in the color rating method; therefore, the evaluation result 


in this study will not be an alternative evaluation for the KC-X program. 


139 “The USAF’s KC-X Aerial Tanker RFP.” 


140 Boeing, Boeing’s Protest File (publicly available part), 
http://www.boeing.com/ids/globaltanker/pdf/executive_summary.pdf (accessed March 18, 2008). 
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V. MODEL CONSTRUCTION 


A. INTRODUCTION 


In this study, two models that have been built via the use of Microsoft Excel are 
presented in two versions to show different characteristics of two types of methods that 
can be used in the source selection phase in government acquisitions. These methods are 
the color rating method, which is the current method used by the Air Force, and the 
quantitative weighted sum method. In the study, the authors have added a numerical 
structure to the color rating method to be able to perform their analysis. Since it is not 
possible to compare numerical and adjective information and make an analysis, the 
authors have converted the color/adjectival ratings into numerical points using a way that 


will be thoroughly explained in this chapter. 


The models have been established using some data from the Air Force’s current 
KC-X Tanker Replacement Program and proposals of its two offerors, Boeing and 
Northrop/Airbus. 


In version one of the authors’ models, they used exact color and adjectival ratings 
given in the KC-X solicitation and determined the ranges that are used to assign 
numerical points to factors and subfactors centered on a basic logic. On the other hand, in 
the second version, the authors eliminated some color and adjectival ratings and changed 
the structures of the ranges to be able to respond to some complicated needs. The details 


about both versions will be provided in the authors’ model building explanations. 
B. MODEL BUILDING OF VERSION ONE 


In model building for version one, the authors will use a model for color rating 
evaluations and will use another model for weighted sum method evaluations. The idea 


behind these models is to show the difference between two techniques for the same 
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purpose and also to show how the result will be affected. The steps that compose the 


version one models with detailed explanations will start with the color rating model and 


after finishing it completely, the weighted sum method model will be explained. 


1. The Model Using the Color Rating Method 


The model building for the color rating method will consist of the following eight 


steps: 


1. Determining factors and subfactors 


2. Determining relative importance of factors and subfactors 


Evaluating the proposed systems and giving ratings to factors and 


subfactors 


4. Assigning weights to factors and subfactors based on their relative 


importance 


SoS 


Determining the ranges of ratings and the ranges' midpoints 
Assigning the related values for those ratings 
Calculating weighted ratings and final ratings 


Comparing both offeror's final ratings and deciding on the winner 


a. Determine Factors and Subfactors 


Factors and subfactors are the discriminators against which each offeror’s 


proposal is evaluated. Factors without any subfactors as well as factors with several 


subfactors are possible in big government acquisitions. In the KC-X program, the factors 


and subfactors were presented in the solicitation as follows: 


* Factor 1: Mission Capability 


Subfactor 1: Key System Requirements (KSR) 
Subfactor 2: System Integration and Software (SIS) 
Subfactor 3: Product Support (PS) 

Subfactor 4: Program Management (PM) 


Subfactor 5: Technology Maturity and Demonstration 
(TMD) 


e Factor 2: Proposal Risk 


e Factor 3: Past Performance 
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e Factor 4: Cost/Price 
e Factor 5: Integrated Fleet Aerial Refueling Assessment (IFARA) 


In the proposal risk assessment, the offerors are evaluated at the mission 
capability subfactor level using the first four subfactors, which are key system 
requirements, system integration and software, product support, and program 
management. Therefore, these four factors are considered as subfactors in the proposal 


risk assessment in the authors’ models. 


Furthermore, a cost/price risk evaluation as well as life cycle cost is 
included in the cost/price factor assessment. Hence, there are two subfactors, life cycle 


cost and cost risk, under this factor. 


In the integrated fleet aerial refueling assessment, the “fleet effectiveness 
value”, which is the standalone value that will be reported to the Source Selection 
Authority (SSA) according to the solicitation, will be the only consideration point on this 


factor. 


After these explanations the final form for the factors and subfactors will 


be as follows: 


® Factor 1: Mission Capability 
e Subfactor 1: Key System Requirements (KSR) 
e Subfactor 2: System Integration and Software (SIS) 


e Subfactor 3: Product Support (PS) 


° Subfactor 4: Program Management (PM) 
° Subfactor 5: Technology Maturity and Demonstration 
(TMD) 
° Factor 2: Proposal Risk 
e Subfactor 1: Key System Requirements (KSR) 
e Subfactor 2: System Integration and Software (SIS) 


° Subfactor 3: Product Support (PS) 
e Subfactor 4: Program Management (PM) 


e Factor 3: Past Performance 
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e Factor 4: Cost/Price 
e Subfactor 1: Life Cycle Cost 
e Subfactor 2: Cost Risk 
* Factor 5: Integrated Fleet Aerial Refueling Assessment (IFARA) 


b. Determine Relative Importance of Factors and Subfactors 


In the KC-X solicitation, the relative importance of factors and subfactors 
is given using narrative explanations and terms like “significant”, “more important,” 
“equal,” or “less important”. As an important point, in the color rating method, a 
mathematical differential between factors and subfactors using terms like “twice as 


important as” is not assigned. !4! 


The relative importance of factors and subfactors is provided in the KC-X 
solicitation as follows: 
The Mission Capability, Proposal Risk, and Past Performance evaluation 
factors are of equal importance and individually more important than 
either Cost/Price or IFARA evaluation factors individually. The IFARA is 
equal in importance to Cost/Price. Within the Mission Capability factor, 
the five (5) subfactors are listed in descending order of relative importance 
from 1 to 5. In accordance with FAR 15.304(e), the Mission Capability, 
Proposal Risk, Past Performance, and IFARA evaluation factors, when 


combined, are significantly more important than Cost/Price; however, 
Cost/Price will contribute substantially to the selection decision. !42 


As stated above, mission capability, proposal risk, and past performance, 
and on the other side cost/price and IFARA, are equally important, whereas the first three 
factors are individually more significant than either cost/price or IFARA. Also, the order 
of relative importance for the mission capability subfactors is as follows: 

1. Key System Requirements (KSR) 

pas System Integration and Software (SIS) 

3. Product Support (PS) 


141 Slate, “Best Value Source Selection,” 52. 
142 USAF’s FA8625-07-R-6470 numbered Solicitation (RFP) Document, Section M, 2. 
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4. Program Management (PM) 
3: Technology Maturity and Demonstration (TMD) 


On the other hand, under the cost/price factor, it has been assumed that life 
cycle cost is more important than the cost risk assessment when its real importance as 


stated in the solicitation is considered. 


c. Evaluate the Proposed Systems and Assign Ratings to Factors 
and Subfactors 


In this phase of the authors’ model building, they have mainly used the 
data they gathered from different sources, i.e., bidders’ web sites. In this way, the 
authors’ goal is to maintain realism throughout their models. Color and adjectival ratings 
are assigned to factors and subfactors for each offeror based on the merits of their 


proposals. 


At this point, it should be stated that the Source Selection Authority (SSA) 
is supposed to make its decision based on these color and adjectival ratings after this step. 
However, because it is so complicated to make a decision based on these ratings and 
consistency is so important while evaluating both offerors, the SSA should use a type of 
numerical method. Based on this logic, the authors have built the rest of their color rating 


model using some numerical ways. 


d. Assign Weights to Factors and Subfactors Based on Their 
Relative Importance 


In the authors’ models, they have arbitrarily assigned some specific 
weights to factors and subfactors to be able to reflect their real importance levels as given 
in the RFP. Without finding an appropriate way to reflect these importance levels, it is 
impossible to be consistent while evaluating both offerors, and assigning numerical 
weights is the most easy and non-complex way to accomplish this goal. As an example, 
because mission capability is more important than cost/price according to the solicitation, 


mission capability should get a higher weight than cost/price. 
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On the other hand, at each level of factors and subfactors, the sum of 
weights should be equal to 100 to maintain consistency. Figure 5 depicts the weights the 


authors assigned. 


KC-X 


Mission Past Performance Proposal Risk Cost/Price IFARA 
Capability 







Xw, = 100 








KSR SIS) PS PM TMD Cost Rusk 
SO Sa 2D Leow =U 60 40 
=w, = 100 Xw, = 100 


KSR SIS PS PM 
oy S10) 15 
Xw, = 100 


Figure 5. The Weights of the Factors and the Subfactors. 


e. Determine the Ranges of Ratings and the Ranges' Midpoints 


In order to combine weights and ratings at the last phase of the evaluation 
and reach an overall rating, each color rating and adjectival rating should get a numerical 
value. Therefore, in the authors’ model building, they have initially established some 
numerical ranges that represent each color and adjectival rating and are set between 


numbers 0 and 100. 


It was stated before that any factor or subfactor except life cycle cost and 
IFARA, which get numerical values, can get any color or adjectival rating based on the 
value an offeror proposes. In order to show which points between 0 and 100 represent 
which color or adjectival rating, a number of ranges should be determined based on the 


number of color or adjectival ratings possible for a factor or a subfactor. 
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In this part of the authors’ study, the ranges for each color or adjectival 
rating are set as equal. For example, when there are four color ratings, each color rating 
will have a range of 25 points on a 100-point scale. Choosing unequal ranges is 


investigated in an upcoming section. 


Additionally, the authors have to assign only one numerical rating to each 
color or adjectival rating and that numerical point should be a number among the 
numbers in the range that rating is associated with. In this study, the authors have 
assigned the midpoints of the ranges to the color and adjectival ratings because a 
midpoint is the most appropriate value that can represent the color or adjectival rating in 


a range. 


The following diagrams in Figure 6 show the ranges and the midpoints for 


the subfactors. 


Mission Capability Subfactors 1-4 Mission Capability Subfactor 5 
RED YELLOW GREEN BLUE RED YELLOW GREEN 
12.5 37.5 62.5 87.5 16.7 50 83.3 
0 25 50 75 100 0 33.3 66.6 100 
Proposal Risk Cost Risk 
HIGH MODERATE LOW HIGH MODERATE LOW 
16.7 50 83.3 
0 33.3 66.6 100 0 | 63330 (8 Ogee O40 


Past Performance 


NO LITTLE UNKNOWN | SATISF. , SIGNIFIC. HIGH l 
8.35 25 41.65 58.35 75 91.65 
0 16.7 33.3 50 66.7 83.3 100 


Figure 6. The Ranges and the Midpoints of the Factors and the Subfactors for Version 
One. 


71 


As a last point, it has been assumed that the cost range is between 105 and 
110 billion dollars because the estimated life cycle cost is about 108 billion dollars. 
Additionally, the IFARA range is between 0.75 and 2.00 because the real IFARA values 
of the offerors are 1.79 and 1.90 and the authors wanted to take into account the values 


below | in this version. 
f Assign the Related Values to the Ratings 


The midpoints determined before for each numerical range were assigned 
to the color and adjectival ratings. In this way, a combination of these points and 


numerical weights will be possible to reach an overall rating for an offeror. 
g. Calculate Weighted Ratings and Final Ratings 


In this phase, weighted ratings are calculated by the multiplication of 
weights and assigned midpoints. For past performance and IFARA, the factors that do not 
have any subfactors, only one calculation is enough to reach the overall factor rating. 
However, for other factors that have subfactors, one more calculation is needed to reach 
the overall factor rating. After having all factor ratings, an overall offeror rating is 


obtained by using the same calculation. 
The basic calculation used in this step is shown below; 
Factor or Subfactor Rating = (Factor or Subfactor Weight) * (Assigned Midpoint) 


Throughout the authors’ study, they have taken advantage of the software 
program Microsoft Excel that can make these kinds of calculations fast and let users 
easily change inputs to see instantaneous changes on outputs. The authors are going to 


explain how they have used Excel during this phase of their study on two examples. 


Before any action, the appearance of the spreadsheet for the mission 


capability subfactors is shown in Figure 7. 
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MISSION CAPABILITY 
SUBFACTORS 


| KSR_ | 
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Figure 7. The Appearance of the Color Rating Method Spreadsheet for the Mission 
Capability Subfactor before Any Action in Version One. 


For the mission capability subfactor key system requirements, after 
evaluation of the proposed system the user enters an assessed color rating into cell C4. As 
soon as the user enters the rating into cell C4, Excel assigns the associated midpoint into 


cell F4 (refer to Figure 8). 


E F G 
MSuaracrons | | ersHTs |RaminG| Varig 
SUBFACTORS RATING 


| KSR_|GREEN 
= 2 ee = 





Figure 8. | The Appearance of the Color Rating Method Spreadsheet for the Mission 
Capability Subfactor after Entering the First Color Rating in Version One. 


To be able to assign the true midpoint associated with any color rating, the 


F4 cell as seen in Figure 8 has the following formula: 


=IF(C4="BLUE",87.5,IF(C4="GREEN",62.5,IF(C4="YELLOW",37.5,[F(C4="RED", 
12.5,0)))) 
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This formula tells Excel to assign 87.5 if the color rating is blue, 62.5 if 
the color rating is green, 37.5 if the color rating is yellow, 12.5 if the color rating is red, 
or 0 for other entries. Since green has been entered into cell C4, Excel assigned 62.5 into 
cell F4. Also, the multiplication of weight (E4) and numerical rating (F4) gives the 
weighted rating. 


After entering all assessed color ratings into cells C4-C8, Excel assigns 
related midpoints into cells F4-F8, which have the same formula above (except F8 
because Technology Maturity and Demonstration (TMD) subfactor does not get a blue 
rating and has different midpoints) with true cell names, and all weighted ratings are 
calculated via the scheme stated above. As a final step, Excel instantaneously adds all 
weighted ratings and the overall numerical rating for the mission capability factor, which 


is 52.08 in this example, is reached (refer to Figure 9). 








A| B | Cc Ip) ET | | H 
1 
2 MISSION CAPABILITY aa WEIGHTED 
3 SUBFACTORS RATING 
4| | KSR__|GREEN 
5| | sis [YELLOW 9. x18 
6| | PS [RED | 
7| | PM [BLUE 
8 | |_TMD [GREEN 
9 
10 


Figure 9. | The Appearance of the Color Rating Method Spreadsheet for the Mission 
Capability Subfactor after Entering All Color Ratings in Version One. 


As a second example, the appearance of the spreadsheet for the cost/price 


factor before any entry is given below in Figure 10. 
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Figure 10. The Appearance of the Color Rating Method Spreadsheet for the Cost/Price 
Subfactors before Any Action in Version One. 


For the cost/price subfactor cost, the user enters the life cycle cost the 
offeror proposes into cell C4. As soon as the user enters the cost data into cell C4, Excel 


assigns the associated numerical rating into cell F4 (refer to Figure 11). 


Due to the fact that the life cycle cost range is between 105 and 110 billion 


dollars, F4 cell has the following formula: 
=IF (C4==0, 0, 2200-(20*C4)) 


This formula tells Excel to assign 0 points into cell F4 whenever there is 
no value assigned and whenever a value is assigned, the formula will use the equation to 
calculate the rating. For instance, if the proposed cost is 110 billion dollars, Excel will 


assign 0 points into cell F4 and 100 points if the proposed cost is 105 billion dollars. 
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Figure 11. The Appearance of the Color Rating Method Spreadsheet for the Cost/Price 
Subfactors after Entering the LCC in Version One. 


For the cost/price subfactor risk, after evaluation of the proposed system, 
the user enters the assessed adjectival rating into cell C5. As soon as the user enters the 
rating into cell C5, Excel assigns the associated midpoint into cell F5 (refer to Figure 12 


below). 
ves, 
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Figure 12. The Appearance of the Color Rating Method Spreadsheet for the Cost/Price 
Subfactors after Entering All Ratings in Version One. 


To be able to assign the true midpoint associated with any adjectival 


rating, the FS cell has the following formula: 
=IF (C5="HIGH",16.7,1F(C5="MODERATE",50,IF(C5="LOW",83.3,0))) 


This formula tells Excel to assign the value of 16.7 if the adjectival rating 
is high, 50 if the rating is moderate, 83.3 if the rating is low, or 0 for all other entries. 
Since low has been entered into cell C5, Excel has assigned 83.3 to cell F5. Also, the 
multiplication of weight (E5) and numerical rating (F5) gives the weighted rating. As a 
final step, Excel instantaneously adds all weighted ratings, and the overall numerical 


rating for cost/price factor, which is 69.32 in this example, is obtained. 


Finally, after all factor ratings are obtained and weighted factor ratings are 
computed using factor weights assigned in step (d), an overall offeror rating is reached by 
adding all weighted factor ratings. The final appearance of an example spreadsheet in 


which the overall offeror rating is 58.2025 is given below in Figure 13. 





pa ess peed 


Mission Capabilit 0.250 52.08 13.02 
Proposal Risk 0.250 63.32 15.83 


0.250 58.35] 14.5875 
0.125 69.32 8.665 
IFARA 0.125 48.80 47 


OVERALL OFFEROR RATING: 





Figure 13. Overall Offeror Ratings for Version One. 
76 


h. Compare Both Offeror's Final Ratings and Decide the Winner 


As the last phase, the two offerors are compared using their overall 
numerical ratings and the offeror that has higher point wins the competition. Thus, the 


contract is awarded to that offeror. 


An example snapshot of the version one color rating spreadsheet as a 


whole is presented in the Appendix part of the study. 
Zz The Model Using the Quantitative Weighted Sum Method 


This method consists of the same steps like the color rating method except steps 
(e) and (f), because numerical ratings instead of color or adjectival ratings are used in the 
quantitative weighted sum method and thus there is no need to determine any range and 
the associated midpoint. In this method the proposed systems of the offerors are given 
numerical ratings, not color or adjectival ratings, between 0 and 100 for each factor and 
subfactor, and after calculating the weighted ratings using assigned weights, the sum of 


the weighted numerical ratings provides the overall rating for each offeror. 


Hence, the steps that compose the quantitative weighted sum method are as 


follows: 

1. Determine factors and subfactors 

2: Determine relative importance of factors and subfactors 
Evaluate the proposed systems and give numerical ratings to factors and 
subfactors 

4. Assign weights to factors and subfactors based on their relative 
importance 

5. Calculate weighted ratings 

6. Compare both offeror's final ratings and decide the winner 


The important point to emphasize about this model is the fact that any number 
between 0 and 100 can be assigned to any factor or subfactor. Therefore, when it is 
considered that in the color rating method, only a few numerical values can be assigned 
to each factor or subfactor, the quantitative weighted sum method is more sensitive to 
small differences between offerors. For instance, although two offerors that propose 
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products with little differences and one of which is better than the other get most likely 
the same color ratings, when the quantitative weighted sum method is used, the 
difference between offerors will be noticeable because of dissimilar assigned numerical 


ratings. 


In this model building phase, the authors have once again taken advantage of 
Microsoft Excel and its user friendly spreadsheets. One example is going to be used to 


explain how Excel was used in this phase of the authors’ study. 


For the proposal risk factor, the appearance of the spreadsheet before any action is 


shown in Figure 14. 
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Figure 14. The Appearance of the Weighted Sum Method Spreadsheet for the Proposal 
Risk Factor before Any Action in Version One. 


For the proposal risk subfactor key system requirements, after evaluation of the 
proposed system, the user enters an assessed numerical rating into cell C4. As soon as the 
user enters the rating into cell C4, Excel computes the weighted rating by multiplying 
that rating with the related weight and assigns that weighted rating into cell E4 (refer to 
Figure 15 below). 
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Figure 15. The Appearance of the Weighted Sum Method Spreadsheet for the Proposal 
Risk Factor after Entering the First Rating in Version One. 


After entering all the assessed numerical ratings into cells C4-C7, Excel calculates 
all weighted ratings and assigns them into cells E4-E7. As a final step, Excel 
instantaneously adds all weighted ratings and the overall numerical rating for the 


proposal risk factor, which is 58.95 in this example, is determined (see Figure 16). 
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Figure 16. The Appearance of the Weighted Sum Method Spreadsheet for the Proposal 
Risk Factor after Entering All Ratings in Version One. 


An example snapshot of the version one weighted sum method spreadsheet as a 


whole is presented in the Appendix part of the study. 
C. MODEL BUILDING OF VERSION TWO 


In this version of the authors’ model, the authors have made some changes on the 
color and adjectival ratings as well as the ranges that are used to determine the numerical 


midpoints for the ratings. 
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First of all, the authors have eliminated some color or adjectival ratings that 
reflect the lowest values for factors or subfactors. For instance, the red color rating for the 
mission capability subfactors has been discarded because the authors thought it is 
reasonable to assume that an offeror who gets a red for any mission capability subfactor, 
which is one of the most critical requirement of a program, does not deserve to compete 
any more for the contract because of its unacceptable offer. As another example, a high 
adjectival rating for the proposal risk factor and cost risk subfactor has also been 


eliminated to disqualify any offeror who puts high risk on the program’s goals. 


In addition to eliminating some ratings, the authors have extended or narrowed 
the numerical ranges to which color and adjectival ratings are related. The logic behind 
this is actually the desire to value an offeror who proposes to provide more valuable 
product to the government. For instance, narrowing the numerical range of blue, which is 
the highest color rating for the mission capability subfactors, increases the likelihood an 
offeror who really deserves and gets that rating will win the contract. On the other hand, 
extending the numerical range of green through high points makes it difficult to get a 


blue rating. 


The following diagrams depicted in Figure 17 show the new ranges and the 


midpoints for the factors and subfactors. 
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Mission Capability Subfactors 1-4 Mission Capability Subfactor 5 


YELLOW GREEN BLUE YELLOW GREEN 
12.5 45 82.5 30 80 
0 25 65 100 0 60 100 
Proposal Risk Cost Risk 
MODERATE LOW MODERATE Low 
30 80 
0 za 60 = 100 0 60 100 


Past Performance 


LITTLE UNK. SATISFACTORY SIGNIFICANT HIGH 


7.5 22.5 45 72.5 92.5 
0 15 30 60 85 100 


Figure 17. The Ranges and the Midpoints of the Factors and the Subfactors for Version 
Two. 


As a last point, in this version of the authors’ study, the IFARA range is 
considered between 1.00 and 2.00 to disqualify any offeror whose proposed system gives 
a fleet effectiveness value of less than one because it has been assumed that a tanker 
aircraft which is not even as effective as the KC-135 can not meet government best value 


requirements. The remainder of each model is completely as same as the initial ones. 
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VI. ANALYSIS AND RESULTS 


A. INTRODUCTION 


In the KC-X Tanker Replacement Program, which is the most significant 
procurement program in the Air Force recently, the Air Force has used its current source 
selection method, color rating, and the Airbus/Northrop Grumman team won the contract. 
After debriefings presented to both offerors, the fact has emerged that the ratings of both 
offerors are almost same and they are very similar where they are not exactly the same. 
Consequently, long debates have begun between Boeing and the Air Force and Boeing 
has filed a protest with the Government Accountability Office (GAO) asking the agency 


to review the award decision. 


What if the Air Force had used a numerical method to determine the winner, not 
the color rating method? Could Boeing have won the contract and be the USAF’s choice 


to build the nation’s next generation tanker plane? 


This is the question that the authors are going to answer in this chapter based on 
the models explained previously. In this chapter, two versions of the authors’ models will 
be presented and it will be demonstrated on two versions that if the Air Force had used 
the weighted sum method to determine the winner, Boeing might have won the contract. 
This result is based on hypothetical data that would have resulted in the actual 
color/adjectival ratings given in Figure 4. This result will prove that Air Force’s source 
selection process using the color rating method has some subjectivity and using a 
numerical method might give completely different results, thus leading another offeror to 


end up as the winner. 


One important point to state is that while assigning numerical points to each 
evaluation subfactor in the quantitative weighted sum method, the authors have assigned 
values among the numbers falling in the range of the associated color or adjectival rating 
to demonstrate that Boeing could have had the better offer. For instance, in version one, 


the mission capability assessment for Boeing when using the quantitative weighted sum 
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method; the authors have assigned 93 (which is a pretty high rating) to the key system 
requirements subfactor. They did so because they had to select a number that would be 
able to represent the blue rating and, therefore, a number between 75 and 100, which is 


the numerical range for the blue color rating in version one. 
B. THE ANALYSIS OF VERSION ONE 


Version one of the authors’ model has been built based on a basic logic in which 
the ranges of color and adjectival ratings are evenly distributed and all color/adjectival 
ratings are taken into account during the evaluations. Version two, which will be 


analyzed later in the chapter, excludes some of the color/adjectival ratings. 


In this part, first of all, each evaluation factor will be analyzed separately for both 
offerors using both methods and overall factor ratings will be obtained. After that, overall 
offeror ratings will be calculated and the differences between those overall ratings will be 


examined. 


There are four tables in each factor assessment part below. The first tables 
demonstrate Boeing’s color ratings assigned to each subfactor under each factor for the 
color rating method, and the second tables show its numerical ratings assigned to the 
same subfactors for the quantitative weighted sum method with consideration given to the 


numerical ranges of the color ratings assigned in the color rating method. 


On the other hand, the third tables demonstrate Airbus’s color ratings assigned to 
each subfactor under each factor for the color rating method, and the fourth ones show its 
numerical ratings assigned to the same subfactors for the quantitative weighted sum 
method considering the numerical ranges of the color ratings assigned in the color rating 
method. Besides, overall factor ratings for each offeror calculated using both methods are 


shown below in tables. 


As explained before in Chapter V, in models using the color rating method the 
authors have converted the color ratings into numerical ratings using range midpoints in 
order to be able to determine an overall score to compare the color rating method and the 


weighted sum method. In the models using the weighted sum method, the numerical 
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values for each factor and subfactor were selected by the authors, because true values 
were not available, to demonstrate how other values within the same color ranges could 
yield different overall results. Also, all weights used to show importance levels of factors 
and subfactors were selected by the authors with consideration given to the importance 


level requirements in the RFP. 
1. Mission Capability Factor Assessment 


The tables demonstrating both offerors’ ratings obtained by using both methods 


for the mission capability factor are provided below in Tables 7 and 8. 


Using Color Rating (Table-1) Using WSM Rating (Table-2) 


MISSION CAPABILITY WEIGHTD 

SUBFACTORS iia Peace RATINGS 
[KS [BLUE 26.250 
[SIS [GREEN 15.625 


SUBFACTORS RATINGS 





Table 7. | Mission Capability Factor Assessment for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


MISSION CAPABILITY WEIGHTD] [MISSION CAPABILITY WEIGHTD 
HT| RAT 

SUBFACTORS RATINGS supractors | CH") eatincs 

26.250 24.30 

15.625]{_ SIS | 600.25 15.00 


Pps | 90] 0.20 | 18.00 





Table 8. | Mission Capability Factor Assessment for Airbus. 


As seen on the tables above, while Boeing and Airbus both get 77.08 as their 
overall mission capability factor rating when the color rating method is used because they 


have the same color ratings for these subfactors, Boeing gets 81.15 and Airbus gets 74.50 
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when the numerical method is used. This result demonstrates that although both offerors 
received the same color ratings, Boeing could possibly have proposed a technically better 


product to the government. 
2. Proposal Risk Factor Assessment 


Tables 9 and 10 show proposal risk assessment results of both offerors obtained 


via both methods. 


Using Color Rating (Table-1) Using WSM Rating (Table-2) 


PROPOSAL RISK WEIGHTD 
SUBFACTORS ee es RATINGS 
29.155 
[ SIS |MODERATE 15,000 


SUBFACTORS RATINGS 





Table 9. Proposal Risk Factor Assessment for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


PROPOSAL RISK WEIGHTD PROPOSAL RISK WEIGHTD 
HT 

supractors | CHT RATINGS SUBFACTORS RATINGS 

29.155 29.75 

| SIS_ [MODERATE 15.000 17.10 





Table 10. | Proposal Risk Factor Assessment for Airbus. 


For the proposal risk factor, because both offerors have received the same 
adjectival ratings, they get 73.31 as their overall proposal risk factor rating. However, 
when the quantitative weighted sum method is used, while Airbus gets 78.10, Boeing gets 
a higher score of 83.50. This result demonstrates that although both offerors received the 
same color ratings, Boeing could have proposed a product to the government which is 


less risky for the program’s objectives. 
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3. Past Performance Assessment 


Both offerors’ ratings obtained by using both methods for the past performance 


factor are provided below in Tables 11 and 12. 






Using Color Rating (Table-1) Using WSM Rating (Table-2) 
PAST PERFORMANCE]RATING RATING 
SATISFACTORY [58.35 
58.35 


Table 11. Past Performance Assessment for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


PAST PERFORMANCE [RATING PAST PERFORMANCE] RATING 
SATISFACTORY 58.35 





Table 12. Past Performance Assessment for Airbus. 


For the past performance factor, both offerors have received the rating of 
‘satisfactory’ which results in the same score for both offerors when the color rating 
method is used. However, Boeing gets a higher point value when the numerical method is 


used because of the assumption that its past performance is better than Airbus’s. 


4. Cost/Price Factor Assessment 


Below are the tables showing the cost/price factor assessment results obtained 


using both methods of both offerors (Tables 13 and 14). 


Using WSM Rating (Table-2) 


COST/PRICE WEIGHTD 
SUBFACTORS ee RATINGS 


Using Color Rating (Table-1) 


COST/PRICE WEIGHTD 
supractors _|“CHT/RATNG RATINGS 
COST | 108.044 | 0.60 | 39.12 23.472 COST | 108.044 | 0.60 | 39.12 23.472 





Table 13. Cost/Price Factor Assessment for Boeing 
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Using Color Rating (Table-3) Using WSM Rating (Table-4) 


COST/PRICE WEIGHTD| COST/PRICE WEIGHTD 
WGHT| RATNG 
SUBFACTORS WoHT] RATNG| RATINGS subractors |“CHTRATNG! earings 


| COST | 108.01] 0.60 | 39.8] 23.88 | COST | 108.01 | 0.60 | 39.80] 23.88 





Table 14. Cost/Price Factor Assessment for Airbus. 


The tables above show that when the color rating method is used, Airbus gets a 
considerably higher score than Boeing in regards to the cost/price factor. However, when 
the quantitative weighted sum method is used, the difference between each offeror’s 
cost/price factor ratings substantially shrinks because of selecting a high point (58.00) 
among points falling in the ‘moderate’ risk rating numerical range (i.e., 33-66) and 
assigning it as the Boeing’s numerical rating for the cost risk subfactor. The assumption 
for assigning a high point value representing the ‘moderate’ risk rating to Boeing is that it 
might have proposed a life cycle cost that put some risk on the program’s goals and that 


risk was slightly more than what is allowed in order to get a ‘low’ risk rating. 
5. Integrated Fleet Aerial Refueling Assessment (IFARA) 


Below in Tables 15 and 16 are the evaluations of the IFARA factor for both 
offerors using both methods. Since there is not any subfactor under the IFARA factor and 
no color or adjectival ratings have been used in this factor evaluation, no difference can 
be observed between each offeror’s ratings. Airbus gets a higher point value than Boeing 


because of its higher fleet effectiveness value using both methods. 






Using Color Rating Using Numerical Rating 
IFARA | RATING IFARA | RATING 


Table 15. Integrated Fleet Aerial Refueling Assessment (IFARA) for Boeing. 
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Using Color Rating Using Numerical Rating 




















Table 16. Integrated Fleet Aerial Refueling Assessment (IFARA) for Airbus. 


6. Conclusion 


The tables presenting all weighted subfactor ratings and calculated overall offeror 


ratings for both methods and offerors are provided below in Tables 17 and 18. 


Using Color Rating Using WSM Rating 


WEIGHTD WEIGHTD 
WGHT | RATNG RATING FACTORS WGHT | RATNG RATING 
0.250 77.08 19.27 Mission Capability 0.250 81.15 20.28 
0.250 73.31 18.32 Proposal Risk 0.250 83.50 20.87 
0 0 


Past Performance i475 
Cost/Price 5.83 
IFARA 10.40 


OVERALL OFFEROR RATING: 68.01 OVERALL OFFEROR RATING: 72.14 





Table 17. Overall Scores for Boeing. 


Using Color Rating Using WSM Rating 


WEIGHTD WEIGHTD 
FACTORS WGHT | RATNG RATING FACTORS WGHT | RATNG RATING 


Mission Capability 0.250] 77.08 19.270 Mission Capability | 0.250] 74.50 18.625 
Proposal Risk 0.250} 73.31 18.328 Proposal Risk 0.250{ 78.10 19.525 


OVERALL OFFEROR RATING: OVERALL OFFEROR RATING: 





Table 18. Overall Scores for Airbus. 


The results are really noteworthy considering the importance of the KC-X Tanker 
Replacement Program. As easily seen above, there are two different winners to whom the 
contract should be awarded when using two kinds of methods. When using the color 


rating method, Airbus gets a score of 70.835 and Boeing gets a score of 68.019, which 
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means Airbus deserves to get the contract. On the other hand, when the quantitative 
weighted sum method is used, Airbus gets an overall score of 70.135 and Boeing gets 


72.147, so the winner should be Boeing. 


The implication under these results is that when there are little but important 
differences between offerors, the color rating method is potentially incompetent in 
reflecting those differences. However, the quantitative weighted sum method is capable 


of handling those little details and providing more accurate results. 


The snapshots of all the models and a comparison chart for version one can be 


found in the Appendix part of the study. 
C, THE ANALYSIS OF VERSION TWO 


As explained in the previous chapter, version two differs from version one based 
on some changes of the color/adjectival ratings and the ratings’ numerical ranges. 
Initially, the color/adjectival ratings that reflect the lowest values for factors/subfactors 
are eliminated and, then, the numerical ranges to which the color and adjectival ratings 
are related to are extended or narrowed. For instance, the high adjectival rating for the 
proposal risk factor and cost risk subfactor has been eliminated to disqualify any offeror 
who puts high risk on the program’s goals, and the range of the low risk rating has been 
narrowed to value the offeror who proposes a less risky product to the government. More 
detailed explanations about the logic behind these changes were provided in the previous 


chapter. 


In this part, first of all, each evaluation factor will be analyzed separately for both 
offerors using both methods. After overall factor ratings are obtained, overall offeror 
ratings will be calculated and the differences between those overall ratings will be 


discussed. 
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There are four tables in each factor assessment part below. The first tables 
demonstrate Boeing’s color ratings assigned to each subfactor under each factor for the 
color rating method, and the second tables show its numerical ratings assigned to the 
same subfactors for the quantitative weighted sum method with consideration given to the 


numerical ranges of the color ratings assigned in the color rating method. 


On the other hand, the third tables demonstrate Airbus’s color ratings assigned to 
each subfactor under each factor for the color rating method, and the fourth ones show its 
numerical ratings assigned to the same subfactors for the quantitative weighted sum 
method considering the numerical ranges of the color ratings assigned in the color rating 
method. Besides, overall factor ratings for each offeror calculated using both methods are 


shown below the tables. 


At this point, the authors want to remind the reader that the difference between 
the analysis tables for versions one and two is that in version two the color rating method 


rating values reflect new midpoints for the color ratings. 
1. Mission Capability Factor Assessment 


The tables showing the mission capability factor assessment results of both 


offerors that were obtained using both methods are provided below in Tables 19 and 20. 
Using Color Rating (Table-1) Using WSM Rating (Table-2) 
MISSION CAPABILITY WEIGHTD]| | MISSION CAPABILITY WEIGHTD 
HT 
supractors — |WCHT/RATNG) 2 tings SUBFACTORS RATINGS 





Table 19. Mission Capability Factor Assessment for Boeing. 
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Using Color Rating (Table-3) Using WSM Rating (Table-4) 


MISSION CAPABILITY WEIGHTD]| |MISSION CAPABILITY wont WEIGHTD 
WGHT 

supractors | WGHT/RATNG! 2 ings SUBFACTORS RATINGS 

BLUE 82.50 24.75 24.30 

GREEN 45.00 11.25} | SIS [600.25 _ | 15.00 


rps | __90 | 020 | 18.00 





Table 20. Mission Capability Factor Assessment for Airbus. 


Since Boeing and Airbus have the same color ratings for the mission capability 
subfactors, they both get 67.25 as their overall factor rating when the color rating method 
is used. However, when the quantitative weighted sum method is used, Boeing gets a 
score of 81.15 and Airbus gets 74.50. The answer to how Boeing may get a higher score 
when the numerical method is used is hidden behind the features of the quantitative 
weighted sum method. Due to the fact that it is more capable of showing little differences 
between offerors, the final scores may be very different although they may be exactly the 
same when the color rating method is used. In this particular case, Boeing might have 


proposed a more technically capable product to the government. 
2, Proposal Risk Factor Assessment 


The proposal risk factor assessment tables that demonstrate all scores for both 


offerors and methods are shown below in Tables 21 and 22. 


Using Color Rating (Table-1) Using WSM Rating (Table-2) 


PROPOSAL RISK WEIGHTD PROPOSAL RISK WEIGHTD 

SUBFACTORS a aaah RATINGS SUBFACTORS bara RATINGS 
80,00[ 28.00 32.90 
17.10 


sis [MODERATE] 030 [ 30.00] 9.00 





Table 21. Proposal Risk Factor Assessment for Boeing. 


92 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


PROPOSAL RISK WEIGHTD PROPOSAL RISK WEIGHTD 

supractors |“CHT! RATING) 2 stings suBractors |“ CHT) patinGs 
80.00 28.00 29.75 
| SIS_ [MODERATE 30.00] ____9.00 17.10 





Table 22. Proposal Risk Factor Assessment for Airbus. 


These results are very similar to the previous ones. Although both offerors get the 
very same score, which is 65, for the proposal risk factor when the color rating method is 
used, Boeing’s score is higher than the Airbus score when the quantitative weighted sum 
method is used. The possible reason for this is that Boeing proposed a product that puts 
lower risk on the schedule, cost and performance goals even though they received the 


same adjectival ratings. 
3. Past Performance Assessment 


The tables of the past performance assessment for both offerors using both 


methods are given below in Tables 23 and 24. 









Using Color Rating (Table-1) Using WSM Rating (Table-2) 
|PAST PERFORMANCE |RATING PAST PERFORMANCE| RATING | 


nn 


Table 23. Past Performance Assessment for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


PAST PERFORMANCE|RATING| PAST PERFORMANCE] RATING 
SATISFACTORY 





Table 24. Past Performance Assessment for Airbus. 
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For the past performance factor, both offerors have received the rating of 
‘satisfactory’ that causes the same score for both offerors when the color rating method is 
used. However, Boeing gets a higher point value when the numerical method is used 
because of the assumption that its past performance is better than Airbus and almost high 


enough to get a ‘significant’ past performance rating. 
4. Cost/Price Factor Assessment 


Tables 25 and 26 show the cost/price factor assessment results of both offerors 


that were obtained using both methods. 


Using Color Rating (Table-1) Using WSM Rating (Table-2) 


COST/PRICE WEIGHTD 
SUBFACTORS eT eae RATINGS 


COST | 108.044 | 0.60 | 39.12 23.472 


COST/PRICE WEIGHTD 
SUBFACTORS POET RAINS RATINGS 


COST} 108.044 | 0.60 | 39.12 23.472 





Table 25. | Cost/Price Factor Assessment for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


COST/PRICE WEIGHTD COST/PRICE IGHTD 
Fcost [10.01] 060] 3980] 23.88) Cost [rosor om [sono] 23.88 


Table 26. Cost/Price Factor Assessment for Airbus. 














For the cost/price factor, Airbus always gets a relatively higher score than Boeing 
when either method is used because of its ‘low’ cost risk rating. However, as easily seen 
on the tables above, the difference between the overall factor ratings of the offerors 
diminish when the quantitative weighted sum method is used. The assumption that leads 
the authors to reach this result is that while Boeing might have proposed a life cycle cost 
that is not realistic enough to get a ‘low’ risk rating but very close to it, Airbus might 
have proposed a cost that is slightly more realistic than Boeing and is able get a ‘low’ risk 


rating. 
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5. Integrated Fleet Aerial Refueling Assessment (IFARA) 


The evaluations of the IFARA factor for both offerors obtained using both 


methods are given below in Tables 27 and 28. 
Using Color Rating (Table-1) Using WSM Rating (Table-2) 

















Table 27. Integrated Fleet Aerial Refueling Assessment (IFARA) for Boeing. 


Using Color Rating (Table-3) Using WSM Rating (Table-4) 


IFARA | RATING IFARA | RATING 
Pst 90 










Lo] 90) 
| 90) 


Table 28. Integrated Fleet Aerial Refueling Assessment (IFARA) for Airbus. 


Since the IFARA assessment does not include any color/adjectival rating, there is 


no difference between each offeror’s ratings. Airbus gets a higher point score than 


Boeing because of its higher fleet effectiveness value using both methods. 


6. Conclusion 


The tables presenting all weighted subfactor ratings and calculated overall offeror 


ratings for both methods and offerors are provided below in Tables 29 and 30. 


Using Color Rating Using WSM Rating 


WEIGHTD 
FACTORS WGHT | RATNG 





Table 29. Overall Scores for Boeing. 
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Using Color Rating Using WSM Rating 


WEIGHTD WEIGHTD 
FACTORS WGHT | RATNG RATING FACTORS RATING 


Mission Capability 16.813 Mission Capability 0.250] 74.50 18.625 
0 


5 





Table 30. Overall Scores for Airbus. 


Version two of the authors’ model, which suggests a different way to evaluate 
both offerors, results in the same significant result as was obtained in the first version. In 
this version, although Airbus gets the higher overall score, which is 62.548, when the 
color rating method is used, it gets a lower overall score, which is 69.885, when the 
quantitative weighted sum method is used. Therefore, there is a change in the winner and 


Boeing gets the contract when the numerical method is used. 


This part of the authors’ study supports their argument that the quantitative 
weighted sum method is more competent than the color rating method in reflecting the 


little but important differences between offerors. 


The snapshots of all the models and a comparison chart for version two can be 


found in the Appendix part of the study. 


In the following chapter, three different factors will be evaluated to understand 
their effects on the results via sensitivity analysis. Sensitivity analysis will be performed 
to demonstrate how these results are impacted by changing the choice of weights. In 
addition, how susceptible these results are to the selection of the rating values with the 


color ranges will be explored. 
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VUl. SENSITIVITY ANALYSIS 


A. INTRODUCTION 


In this chapter, the authors will investigate how modeling choices might affect the 
results that were presented in the previous chapter. In the previous chapter, the authors’ 
study results support their argument that the quantitative weighted sum method is more 
competent than the color rating method in reflecting the little but important differences 
between offerors. The color rating method is potentially incompetent in reflecting those 


differences. 


In this part of the study, sensitivity analysis will be done to understand the effects 
and limits of ratings (within color ranges) and weights in the models. Three different 
forms of sensitivity analysis will be done. In the first one, numerical ratings for the color 
and adjectival ratings will be changed within their ranges to see the best/worst possible 
overall scores for each alternative in version one of the authors’ model. In the second 
one, the same analysis that will be done in the first sensitivity analysis will be applied to 
the authors’ version two model (with a diminished number of color and adjectival 
ratings). In the third final one, the weights will be changed. While keeping original model 
ratings as constant, the authors will analyze the effects of weights on results. The last 
analysis will be done for each alternative and for the color and weighted sum method 


(WSM) rating techniques. 
B. FIRST SENSITIVITY ANALYSIS 


In the previous chapter, the results show that while Airbus wins the contract with 
the color rating method evaluations, in the WSM evaluation Airbus loses it with 
numerical scores that are in the same color ranges (numerical equivalents of color 
ratings). Based on these results, the authors decided to perform a sensitivity analysis to 
understand the effects of the color rating ranges on the results. The authors will keep the 
color rating ranges as constant and will change the numerical ratings that are equivalent 


to those color ratings in the possible rating ranges. The research question is “How 
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sensitive is the overall WSM score to different numerical ratings within each color 
range?” This will give the authors the best and the worst overall score that each company 
could get using the numerical model of their study (while keeping numerical ratings 


within color ranges for assigned colors). 


The authors used the formula on the following page to do the first part of their 
sensitivity analysis. To calculate the best overall score, the authors needed a 
maximization model for the objective function, and to calculate the worst overall score, 
they needed a minimization model for the objective function. Each (max and min) model 


was implemented twice - one for Boeing and one for Airbus. 


The formulation on the next page is for Boeing in version one of the authors’ 
model. With some little changes, this formula adapted to Airbus so that the same analysis 
could be performed. The following model will be solved once for maximization and the 
results will give the best possible score for Boeing with the assigned color and adjectival 
rating ranges in the first version of the authors’ model. Next, it will be solved for 
minimization and the results will give the worst possible score for Boeing with the 
assigned color and adjectival rating ranges in the first version of the authors’ model. 
Then, the adapted version of the following formula will be solved two more times, once 
for maximization and once for minimization, to reach the results for possible best/worst 


scenario overall scores for Airbus. 


In the model, all the subfactors that are assigned with color and adjectival ratings 
will be the decision variables. The factors and subfactors, those get specific numerical 
ratings like IFARA and life cycle cost, will not be included in the model as decision 
variables. The objective function will be an overall score function that consists of all 
subfactor ratings and their weights, and the constraints will be the max and min values for 
each subfactor with color and adjectival ratings. For example, for KSR, decision variable 
x; can get any value between 76 and 100 (including 76 and 100). Since all the decision 
variables will be in a positive number range, there will not be a non-negativity constraint 
(redundant constraint-this constraint does not add any value to the model). Also, the 


model will be solved as a linear model. 
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MODEL(S) FOR BEST/WORST RATINGS WITHIN COLOR RANGES 
(FOR BOEING IN VERSION 1) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) rating. 
X2 : MC subfactor System Integration and Software (SIS) rating. 

x3 : MC subfactor Product Support (PS) rating. 

x4 : MC subfactor Program Management (PM) rating. 

x5: MC subfactor Technology Maturity and Demonstration (TMD) rating. 
X6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) rating. 
x7:  PResubfactor System Integration and Software (SIS) rating. 

xg :  PRsubfactor Product Support (PS) rating. 

x9: PRsubfactor Program Management (PM) rating. 

x10 :  Cost/Price subfactor Cost Risk (CR) rating. 

x7, : Past Performance (PP) rating. 


Objective Function (Linear Model): 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x44+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 wig + 0125 7 83.2 4 





(0.40 x19 + 0.60 * 39.12) 0.125 


Constraints: 


8 88 588 2S 2S 2 IR 2S fe 2 8 88 IS 2S 2 IR 2S 2S 24S 8 88 2S OIE 2S 2 2S 2S 246 2 8 218 FIC 2S OIC IR 2S 24S 8 8 18 2S 2S 2 2 2g fe 24s 2 88 2S 2S 2S 2k 2k 2g 246 24s 8 288 2 2S 2 2k 2k fe 2k 2s 2s 2 2K 2K 


For Mission Capability; 


x; >= 76 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 51 and x2 <=75 (Subfactor SIS is within green color rating range) 

x3 >= 76 and x3 <=100 (Subfactor PS is within blue color rating range) 

x4 >= 51 and x4 <=75 (Subfactor PM is within green color rating range) 

X5 >= 67 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 OIE 24S 8 2A 24g 8 88 2S 2 88 OIE 24S 18 OIE 24 1S OIE OIE 8 OIE OIE 18 OIE OIE 8 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE 2S 8 OIE 2g 8 OIE 2g 8 2A 2g 28 2k 2S 28 2A 2k 2 2K 2k 2g 2K 2k 2s 2 2k 2s 2K 2K oe ok 
For Proposal Risk; 

X6 >= 67 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 34 and x7 <=66 (Subfactor SIS is within moderate adjectival rating range) 
Xg >= 67 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 67 and x9 <=100 (Subfactor PM is within low adjectival rating range) 

388 IE 24S 8 FAS 24 8 88 2S IR 88 OIE 24S 18 OIE IE IS OIE 24 8 OIE OIE 8 OIE OIE 18 OIE OIE 18 FIC OIE 18 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE 2S 28 OIE 2g 88 OIE 2g 8 OIE 2g 2g 2 2g 2g 2k 2k 2 2K 2k 2g 2K 2k 2s 2K 2K ik ok 
For Cost/Price; 

x10 >= 34 and x9 <=66 (CR is within moderate adjectival rating range) 


28 88 388 28 2S 2S IC 2k 24g 2 8 88 28 2S 2S 2K 2S 2S 24g 8 8 28 2S 2S 2 IR 24g 24S 2 8 288 FES 2S 2S 2IC 2S 2S 28 2 88 218 IS 2S 2K 2k 2g 248 2g 8 288 2S 2S 2S 2k 2g 2k 24s fe 8 28 2S 2 2k 2k 2k 2s 2s 2 28s 2 2K 2K 


For Past Performance; 
X71; >= 51 and x;; <=66 (PP is within satisfactory adjectival rating range) 
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To solve the model, Microsoft Excel 2003’s Solver tool was used and a snapshot 


of the spreadsheet is presented in the Appendix section of this study. 


The model above is adapted to Airbus by changing the IFARA rating from 83.2 to 
92.0 and the life cycle cost (LCC) rating from 39.12 to 39.80 in the objective function for 
overall score (because these results are evaluated directly as numerical, they are placed as 
constant values in the model). Also, changing the ranges for x79 (cost risk subfactor) is 
necessary, because Airbus received a ‘low’ adjectival rating instead of a ‘moderate’ 
rating (Boeing’s CR rating) for this subfactor. The new range for x79 is 67 to 100 instead 
of 34 to 66. The model formulation for Airbus is also presented in the Appendix section 


of this study. 


As expected in the solution of the models, while solving for maximization, all the 
ratings (decision variables) got the highest possible points in their range, and while 
solving for minimization, all the ratings got the lowest possible points in their range. For 
example, for mission capability subfactor KSR, Boeing originally received a blue color 
rating which ranges from 76 to 100 on a 100 scale. When solving for maximization, the 
authors’ model assigned 100 points as the rating for this subfactor, and when solving for 


minimization, the model assigned 76 points. 


The overall scores for Boeing are 78.084 as the maximum and 58.334 as the 
minimum. For Airbus, the scores are 80.935 as the maximum and 61.135 as the 
minimum. The ranges for max/min overall scores are 19.75 for Boeing and 19.80 for 
Airbus. Airbus’s max/min scores are both almost 2.80 point higher than Boeing’s scores, 
but their scores are overlapping for 16.95 points. These results are also presented in Table 


Si. 


VERSION? | BOEING | AIRBUS | DIFFERENCE OF BOEING/AIRBUS 
MAX 78.084 | 80.935 J BST 
MIN 58.334 





Table 31. Summary Results Table for Sensitivity Analysis Version One. 
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The sensitivity of the overall scores for both alternatives is very similar. This is 
because of the fact that while using color and adjectival rating methods for evaluation, 
the scores for the two alternatives are very similar. There are just three differences - one 
in the IFARA, one in the LCC, and one in the cost risk rating. Only three out of the 
thirteen criteria that affect the overall score are different and these are only slightly 
different. In addition, since these three factors are also multiplied by small weights, they 
make even a smaller difference in the overall score. Another important finding from these 
results is the high percentage (almost 85%; 16.95/19.80) of overlap between the ranges of 
possible points for both alternatives. This overlapping area means that it is possible for 
either alternative to win the contract whenever the overall scores for both are in this 


overlapping range. 
C. SECOND SENSITIVITY ANALYSIS 


For the second part of the sensitivity analysis, the authors will follow the same 
methodology as the sensitivity analysis in the first part and apply it to the second model. 
The authors will examine the results of the analysis to understand the effects of color 
rating ranges on the overall scores. They will again keep the color rating ranges as 
constant and will change the numerical ratings that are equivalent of those color ratings in 
the possible rating ranges. In this part, because the authors will be working on their 
second version, they will not use some of the color and adjectival ratings that were 
effective in the first model (like red in the color ratings, 'high' in the risk ratings and 'no 
confidence' in the performance ratings), so the color and adjectival rating ranges will be 


changed. 


The research question is again “How sensitive is the overall WSM score to 
different numerical ratings within each color range (with a diminished number of color 
and adjectival ratings)?” This will again give the authors the best and the worst overall 
score that each company could get using the WSM model of their study (while keeping 
numerical ratings within color ranges for assigned colors). Then, the authors will also 
examine the effects of diminishing the effective number of color and adjectival ratings. 


They will calculate the best and worst overall score in the same way, that is a 
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maximization model for the objective function for the best overall score and a 
minimization model for the objective function for the worst overall score. Each (max and 


min) model will be implemented twice, one for Boeing and one for Airbus. 


The formulation for Boeing in the second part of the sensitivity analysis will have 
the same decision variables, a very similar objective function (changes are highlighted), 
and very similar constraints. The only part that will be different from the formulation for 
Boeing in version one is the ranges that are the numbers in the right-hand side of the 
equations for constraints. The new objective function and the new constraints with the 


new ranges as the right-hand side values of the equations are as follows: 


Objective Function: 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x4+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 x7;+ 0.125 * a+ 





(0.40 x70 + 0.60 * 39.12) 0.125 
Constraints: 


388 OIE 24S 88 OIE 24S 88 2A 2S 18 OIE IE 18 OIE 2S 8 OIE 24S 8 OIE 2S 8 OIE 2S 8 OIE OIE 8 OIE 24 18 OIE 2A 8 OIE 2S 8 OIE IR 8 OIE OIE 8 OIE 2S 8 OIE 2S 28 OIE 2g 8 OIE 2g 28 2A 2k 2 2k 2k 2 2k 2k 2s 2K 2 2s 2K 2K ok ok 


For Mission Capability; 


x; >= 66 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 26 and x2 <=65 (Subfactor SIS is within green color rating range) 

x3 >= 66 and x3 <=100 (Subfactor PS is within blue color rating range) 

x4 >= 26 and x4 <=65 (Subfactor PM is within green color rating range) 

X5 >= 61 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 IE 24S 8 2S 24g 8 88 OIE 2S 18 OIE 24S 18 OIE 24S 18 OIE OIE 8 OIE IE 18 OIE 2S 8 OIE OIE 8 OIE 2S 8 OIE 2S 18 OIE OIE 8 OIE 2A 8 OIE 2S 8 OIE 2g 28 OIE 2g 8 OIE 2S 28 2 2g 8 2S 2k 2 2k 2k 2g 2K 2k 2 2K 2k 2s 2K 2K is ok 
For Proposal Risk; 

X6 >= 61 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 0 and x7 <=60 (Subfactor SIS is within moderate adjectival rating range) 
xg >= 61 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 61 and x» <=100 (Subfactor PM is within low adjectival rating range) 

3S OIE IR 8 FAS 24S 8 48 OIE IR 18 OIE 2S 8 OIE 24 18 OIE IE 8 OIE OIE 8 OIE IE 8 OIE OIE 18 OIE OIE 8 OIE OIE 18 OIE OIE 8 OIE OIE 8 OIE OIE 18 OIE 2S 8 2S 2S 8 OIE 2 8 OIE IR 8 OIE OIC 2 OIC 2g 2g 2K 2k 2 2K 2k 2g 2K 2K 2g ok 
For Cost/Price; 

X10 >= 0 and x79 <=60 (CR is within moderate adjectival rating range) 


8 88 598 IS 2S 2S 2S 24S 24S 2 8 8 2S OIE 2 2 IR fg 246 8 8 28 IS 2S 2 IR 24S 24S 2 8 518 IS 2S 2S 2 IR 2S 248 2 8 2S IS 2S 2 IR 2g 246 2 8 218 2S 2S 2S IR 2k 24k 24s fe 8 288 2S 2S 2k 2k 2k 24s 2k 2 2s 2 2K 2K 


For Past Performance; 
x1; >= 31 and x;; <=60 (PP is within satisfactory adjectival rating range) 
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With some little changes, this formula is adapted to Airbus to be able to do the 
same analysis. The model will be solved twice, once for maximization and once for 
minimization, to reach the results for possible best/worst scenario overall scores for 


Boeing and Airbus. 


To solve the model, Microsoft Excel 2003’s Solver tool is used and a snapshot of 


the spreadsheet is also presented in the Appendix section of this study. 


The model in the second sensitivity analysis is adapted to Airbus by changing the 
IFARA rating from 79 to 90 and the life cycle cost (LCC) rating from 39.12 to 39.80 in 
the objective function for overall score (since these results are evaluated directly as 
numerical, they are placed as constant values in the model). Also, changing the ranges for 
X79 (cost risk subfactor) is necessary, because Airbus received a 'low' adjectival rating 
instead of a 'moderate' rating (Boeing’s CR rating) for this subfactor. The new range for 
x10 is 61 to 100 instead of 0 to 60. The model formulation for Airbus is also presented in 


the Appendix section of this study. 


While solving for maximization, all the ratings (decision variables) got the highest 
possible points in their range and while solving for minimization, all the ratings got the 


lowest possible points in their range. 


The overall scores for Boeing are 74.309 as the maximum and 43.609 as the 
minimum. For Airbus, the scores are 77.735 as the maximum and 48.085 as the 
minimum. The ranges for max/min overall scores are 30.70 for Boeing and 29.65 for 
Airbus. The max/min scores for Airbus are both almost 4 point higher than Boeing’s 
scores, but their scores are overlapping for 26.22 points. These results are also presented 


in Table 32. 


VERSION 2 | BOEING | AIRBUS DIFFERENCE OF BOEING/AIRBUS 
MAX | 74309 | 77735 [8A 


MIN | 43.609 | 48.085 4.476 





Table 32. Summary Results Table for Sensitivity Analysis Version Two. 
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The sensitivity of overall scores for both alternatives is very similar. This is 
because while using the color and adjectival rating methods for evaluation, the scores for 
the two alternatives are very similar. There are just three differences - one in the IFARA, 
one in the LCC, and one in the cost risk rating. Only three out of the thirteen criteria that 
are affecting the overall score are different, and these are only slightly different. In 
addition, since these three factors are also multiplied by small weights, they make an 
even smaller difference in the overall score. Another important finding from these results 
is the high percentage (almost 85%; 26.22/30.7) of overlap between the ranges of 
possible points for both alternatives. This overlapping area means that it is possible for 
both alternatives to win the contract whenever the overall scores for both are in this 
overlapping range. Logically, fewer ranges (rating groups) should cause a larger overlap 
region, supporting the notion that the less sensitive the method, the more likely it could 
undervalue small differences. Since the model’s second version had a smaller number of 
color and adjectival ratings, the evaluation results become even less sensitive to the small 


differences. 
D. THIRD SENSITIVITY ANALYSIS 


For the third part of the sensitivity analysis, the weights that the authors have used 
to reflect the relative importance of factors and subfactors will be changed, while keeping 
original model ratings (version 1 of the model) as constant, to analyze the effects of 
weights on results. In this analysis part, all the weights (for factors and subfactors) will be 
changed to analyze the effects on overall scores. The authors’ model for the third part of 
the sensitivity analysis will be solved for maximization and minimization to understand 
the effective change range for weights. The effects of weight change will be analyzed for 
each alternative four times, which are the following: weights for maximum overall score 
in the color rating method, weights for minimum overall score in the color rating method, 
weights for maximum overall score in the WSM rating method, and weights for 
minimum overall score in the WSM rating method. These four analyses will be done for 


each alternative separately and the formulation will have small changes for each iteration. 
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In the model, all the factor and subfactor weights will be the decision variables. 
The objective function will be an overall score function that consists of all subfactor 
ratings and their weights, and the constraints will be the relative importance of subfactors 
and factors with equality of factor/subfactor weight totals to one for each factor/subfactor 
level. Since all the decision variables will be in a positive number range, there will not be 
a non-negativity constraint (redundant constraint-this constraint does not add any value to 


the model). Also, the model will be solved as a non-linear model. 


MODEL(S) FOR BEST/WORST RATINGS FOR WEIGHT CHANGES 
(FOR BOEING FOR COLOR RATING METHOD IN VERSION 3) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) weight. 
XxX, : MC subfactor System Integration and Software (SIS) weight. 

x; : MC subfactor Product Support (PS) weight. 

x, : MC subfactor Program Management (PM) weight. 

xs: MC subfactor Technology Maturity and Demonstration (TMD) weight. 
x6: Proposal Risk (PR) subfactor Key System Requirements (KSR) weight. 
x7:  PRsubfactor System Integration and Software (SIS) weight. 

xs:  PRsubfactor Product Support (PS) weight. 

x9:  PRsubfactor Program Management (PM) weight. 

X30:  Cost/Price subfactor Life Cycle Cost (LCC) weight. 

x7, :  Cost/Price subfactor Cost Risk (CR) weight. 

X12: Mission Capability, Proposal Risk and Past Performance factors’ weight. 
X;3 :  Cost/Price and IFARA factors’ weight. 


Objective Function (Non-linear Model): 


Maximize Overall Score =[(87.5 x; + 62.5 x2+ 87.5 x3+ 62.5 x4+ 83.3 x5)] Xj2 + 


or [(83.3 X56 +50 x7 + 83.3 Xg t+ 83.3 X9)] Xj2 + 
Minimize 58.35 X12 + 83.2 X13 (50 X11 + 39.12 X10) X13 
Constraints: 


ES 24S 24 8 88 288 IS 2S 2S IR 2S 24S 24 8 8 2S 2S 2 2 OIE 2S 24S 8 88 IS OIE OIC 2 2S 24S 24S 2 8 218 IS 2S 2 2S 2S fe 2S 8 8 2S 2S 2K IR 2g fe fs fg 8 2S 2S 2S 2 2k 2g 246 fe 2s 288 2 2 2K 2K 2 2 2 os 2k 


For Mission Capability; 





Xx) > Xp (KSR is more important than SIS) 
X2 > X3 (SIS is more important than PS) 
X3>X4 (PS is more important than PM) 

X44 > X5 (PM is more important than TMD) 
x5>0 (all of the weights are bigger than 0) 
Xp tx. tx34+x¢+X5= 1 (total of weights are equal to 1) 


BRS IS 2S Ie fe 24 8 88 28 2S 2S 2S 2S 2S 2S 24k fe 2 88 218 2S 2S 2S 2S 2S 2S 24 24 fe 8 8 288 2S 28 28 2S 2S OIE 24E fe fe 8 2S 2S 8 2S 2S IE 2S OIE fe fe 2g 8 2S 2 8 21S 2S 2S OIE 2k fe fe fe 8 24g fe 8 28s 2S 2S 2 2k 2k 2k 2 
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For Proposal Risk; 


X6 > X7 (KSR is more important than SIS) 

X7 > Xs (SIS is more important than PS) 

Xg > Xo (PS is more important than PM) 

x9 >0 (all of the weights are bigger than 0) 

Xg6 tx7t+XxXgt+ X= 1 (total of weights are equal to 1) 
oe 2k ie fe of fe of ofc of oft of of of ofc ofc of of of of 2s oie oe of of oie os aie ate afk ofc ofc oft of fe fe ofc ofc oft of of of fe oie oe of of 2k of ofc aie afk afk fe ofc oft oie ofe of ofc of of ok of of fe oe ok ok ok 2k fc 2 2c 2k of ok ok ok 
For Cost/Price; 

x10 > X11 (LCC is more important than CR) 

x11>0 (all of the weights are bigger than 0) 

Xp txy7= 1 (total of weights are equal to 1) 


BAS 8 8 IS IS 2S 24S IS 4S 8 2S 4S 8 2S 2S IE OIE OIE fe 24 4 8 4 8 8 IS IS OIE OIE 2S OIE 24S 2 24 2 8 2S 2S IE IE OIE 24S 24S 24 24E 24S 2 8 28S IS 2S OIE OIE fe 24g 2k fe fe fe 8 288 2S 2S 2S 2k 2g 2k 2k fe fe fe is 2s 2k 2 2K ke 


For Factor Level; 


X72 > X23 (MC,PR and PP are more important than CR and IFARA) 
x13>0 (all of the weights are bigger than 0) 
3x0+2x3= 1 (total of weights are equal to 1) 


The formulation above is the base formulation for the third part of the sensitivity 
analysis. It is the formulation for Boeing’s color rating in version one. In this analysis 
part, the model is no longer linear because of the multiplication of some decision 
variables which causes the authors’ model to become a non-linear model (MC, PR and 
CR subfactor weights are multiplied by factor level weights and then the model becomes 


non-linear). 


In all of the iterations of the model, the decision variables and the constraints will 
not be changed. The only differences will be made in the objective function and it will be 
the change of some constant values for the color rating part for Airbus (the differences 
between the models for Boeing and Airbus are highlighted below), and the change of all 
the constant values for the WSM rating method (for Boeing and Airbus). All three 
objective functions are presented below (not the whole mathematical formulation) and 
also full and separate versions of the formulations for each iteration can be found in the 


Appendix section of this study. 


To solve the model, Microsoft Excel 2003’s Solver tool is used and a snapshot of 


the spreadsheet is also presented in the Appendix section of this study. 
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Objective Function for Airbus Color Rating Weights Change Effects: 





Maximize Overall Score=_ [(87.5 x; + 62.5 x2+ 87.5 x3+ 62.5 x4+ 83.3 x5)| X72 + 
or [(83.3 x6 + 50 x7 + 83.3 xg + 83.3 x9)] x72 + 
Minimize 58.35 X12 +92 Xp3t (83.3 X11 + 39.8 X10) X13 


Objective Function for Boeing Numerical Rating Weights Change Effects: 





Maximize Overall Score=  [(93 x; + 64 x2+ 92 x3+ 63 x4+ 94 x5)] x72 + 
or [(94 x6 + 57 x7 + 97 xg + 94 x9)] X72 + 
Minimize 59 X72 + 83.2 x73+ (58 x77 +3912 X10) X13 





Objective Function for Airbus Numerical Rating Weights Change Effects: 





Maximize Overall Score=  [(81 x;+ 60 x2+ 90 x3+ 58 x4+ 85 x5)] x72 + 
or [(85 x6 + 57 x7 +91 xg + 87 x9)] x12 + 
Minimize 56 X72 +92 X13 (70 X11 + 39.8 X10) X13 





After solving all these iterations to analyze the overall score sensitivity to weight 
change, the overall scores for Boeing are 75.30 as the maximum and 65.04 as the 
minimum for the color rating method and 80.75 as the maximum and 67.38 as the 
minimum for the WSM method. The overall scores for Airbus are 75.97 as the maximum 
and 66.95 as the minimum for the color rating method and 73.44 as the maximum and 


66.18 as the minimum for the WSM method. The results are presented in Table 33; 


COLOR MAX| COLOR MIN | WSMMAX | 
75.30 | 65.04 [my Wed 67.38 
ORIGINAL 68.02 ORIGINAL 72.15 





75.97 66.95 73.44 66.18 
apeee ORIGINAL 70.84 ORIGINAL 70.14 


Table 33. Summary Results Table for Sensitivity Analysis Version Three. 


Also, the change range for overall scores, which is due to weight changes, is very 
similar for Boeing (10.26) and for Airbus (9.03) in the color rating method. However, in 
the WSM method the change range difference is significant, as it is 13.38 for Boeing and 
7.26 for Airbus. Because Boeing’s ratings for most of the subfactors are higher than 
Airbus in WSM method, Boeing is more sensitive to weight changes in WSM method. 
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These results also show the difference between the color rating method and the WSM 
method, as the color rating method is less competent than the WSM method to reflect the 
little changes, the differences in ratings are hide in color ratings and both Boeing and 


Airbus change ranges become similar. 


In this part of the sensitivity analysis, the authors also wanted to analyze the 
overall scores for a special case. They implemented Boeing’s best case scenario weights 
to calculate the overall score for Airbus in the same scenario. The Microsoft Excel 
snapshot of this scenario is shown in the Appendix section of this study. For the color 
rating method, the results were 75.30 for Boeing and 75.56 for Airbus, which means with 
some cost, cost risk rating and IFARA advantage Airbus won with the color rating model 
even with Boeing’s best scenario weights. For the numerical rating, the results were 
80.75 for Boeing and 73.44 for Airbus, which results in a significant advantage for 


Boeing. Table 34 below summarizes these results. 


BOEING'S BEST RESULT RATINGS VS. AIRBUS WITH SAME WEIGHTS 
COLOR NUMERICAL 


BOEING 75.30 80.75 
AIRBUS 75.56 73.44 


Table 34. Summary Results Table for Special Case: Boeing’s Best Case Scenario. 





The authors also implemented the best case scenario weights for Airbus to 
calculate Boeing’s overall score in that scenario. The Microsoft Excel snapshot of this 
scenario is also shown in the Appendix section of this study. For the color rating method, 
the results were 71.00 for Boeing and 75.97 for Airbus, which results in a significant 
advantage for Airbus. For the numerical rating, the results were the same as the previous 
scenario and both alternatives’ best results because the best result weights were the same 
for both alternatives in the numerical method. A summary of these results is shown in 


Table 35. 
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AIRBUS'S BEST RESULT RATINGS VS. BOEING WITH SAME WEIGHTS 
COLOR NUMERICAL 


BOENGD| 71.00 80.75 
AIRBUS 75.97 


Table 35. Summary Results Table for Special Case: Airbus’s Best Case Scenario. 





E. CONCLUSIONS 


The sensitivity analysis to understand the effects of rating change within color 
ranges on both versions of the authors’ study produced identical results. This is due to the 
fact that while using color and adjectival rating methods for evaluation, the scores for two 
alternatives are very similar. There are just three differences - one in the IFARA, one in 
the LCC, and one in the cost risk rating. Only three out of the thirteen criteria that are 
affecting the overall score are slightly different. In addition, since these three factors are 
also multiplied by small weights, they make an even smaller difference in the overall 
score. Another important finding of the results of rating change within color ranges 
effects analysis for the two models is the high percentage of overlap between the ranges 
of possible points for both alternatives. This overlapping area means that it is possible for 
either alternative to win the contract whenever the overall scores for both are in this 
overlapping range. Also, the overlapping area allows subjectivity in the evaluation of 
offeror proposals. The difference between these sensitivity analysis implementations on 
the two versions of the authors’ study is the results of range changes (elimination of some 
ratings lead to redetermination of ranges). Since the ranges get bigger in the second 
version, the effects of rating change get bigger as expected and the amount of subjectivity 


(with growing overlapping ranges) allowed gets increased. 


The sensitivity analysis to understand the effects of weight (factor and subfactor 
weights all together) changes on the overall scores produced interesting results. The 
change ranges for overall scores were very similar for Boeing and for Airbus in the color 
rating method. However, in the WSM method the change range difference was 
significant. These results show how the color rating method is less sensitive than the 


numerical method (less competent to reflect the little changes). 
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The authors finished their sensitivity analysis with a special case scenario 
analysis. Boeing’s best case scenario weights implemented to Airbus and results for the 
color rating method lead to an Airbus award even in Boeing’s best case scenario. In that 
case, it shows that even with higher numerical scores, Boeing will not be able to present 
its better value (better value than Airbus) under the color rating method. In WSM 
method, Boeing’s best case scenario weights implemented to Airbus lead to a significant 
overall score advantage for Boeing. Airbus’s best case scenario weights implemented to 
Boeing resulted with significant advantage for Airbus in color rating method, however 
the results were significantly advantageous for Boeing in WSM method (even in Airbus’s 


best case scenario). 
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VIII. CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY 


The source selection phase of government acquisitions involves many interrelated 
tasks. Even though objective factors and subfactors are used in the source selection 
phase, subjectivity is inevitable in government contracting. This is because the Federal 
Acquisition Regulation (FAR) does not provide any clear guidance to evaluate proposals 
and, thus, the source selection decision is usually made based on some evaluations 
“conducted using any rating method or a combination of methods, including color or 
adjectival ratings, numerical weights, and ordinal rankings.”!43 Besides, proposals are 
evaluated by different personnel and conclusions may differ, even for the same factor, 
significantly.!44 Therefore, considering the high complexity in the nature of government 
acquisitions, the comparison of proposals and source selection decision involve 


judgmental evaluations.!45 


Considering the fact that available resources to the government are becoming 
scarcer, subjectivity should be diminished in contracting and agencies should find ways 
to get the best value from any procurement. Quantitative methods and information 
technology tools can be utilized to augment objectivity in source selection decision 


making. 


The aim of this project is to show the effects of the choice of model (method) 
used, as well as decisions made during the modeling and assessment process. These 
choices may contrast with the fairness and the best value part of government procurement 


processes. Fair to whom and best value for whom were the very first questions in the 


143 RAR 15.305. 
144 Cibinic and Nash, Formation of Government Contracts, 821. 


145 Templin and Noffsinger, “An Assessment,” 38. 
111 


authors’ minds when they started this project. The authors tried to show the effects of 
subjectivity involved in the process of selection by using a simplified current real-life 


program. 


The authors chose the USAF’s KC-X program as a case study for their research 
questions. There are some specific reasons for their choice of that program. The authors 
focused on an Air Force procurement program in which color ratings were used, because 
in their perspective, the Air Force’s color rating is one of the most subjective techniques 
used in source selection. The authors decided to analyze a program that was popular and 
current. The authors thought that they could have a variety of comments from different 
viewpoints. With a current program, they hoped to have more insight information, which 
was publicly available, about the program. After their initial research on the program, the 
authors realized that the two alternatives offered very different specifications and 
capabilities for the same program. This showed the authors the difference of the two 
offerors’ point of view. The authors thought that this program might be a good sample for 
their study because of the potential effects of the choices made for and during the 
evaluation. Another factor that made the choice even more interesting for this study was 
the high potential for the award to be protested. Experts of the defense industry were 
expecting a protest from the loser. Since the proposed KC-X alternatives have very 
different strengths in capabilities, results will not easily satisfy the offeror who loses the 


program award. 


The source selection phase in government contracting involves multi-criteria 
decision making which has to deal with many critical factors such as cost, technical 
capabilities, and the past performance of contractors. The contracting processes used in 
government acquisitions have been built to respond to various government procurement 


needs by evaluating these factors. 


In Chapter II an overview of acquisitions, general summary information about 
acquisition processes was provided. First, contracting methods were discussed and then, 


the evaluation and _ selection processes were defined briefly. Since 
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this project focused on the Air Force’s color rating method, the Air Force source 
selection was examined a little deeper on categories of Air Force acquisitions, steps in 


Air Force source selection, evaluation activities, and award activities. 


Chapter III provided an overview of decision-making support techniques, the 
authors gave some background information about the available techniques to solve a 
multi-criteria problem since the study case (the Air Force’s KC-X program) is a major 
weapon systems procurement and has more than one criterion to determine while trying 
to come up with the best value product for the government. Multi-criteria decision 
analysis (MCDA) offers numerous methodologies and schools of thought for multi- 
criteria modeling. The approaches vary from very simple ones to extremely complex 


optimization models. 


The authors used the weighted sum method (WSM) in their study. They preferred 
a technique that was simple to understand by anyone who has no multi-criteria decision 
making (MCDM) background. WSM is one of the very basic models/techniques used in 
situations where more than one criterion is involved and the relative importance of each 
criterion is different from each other. The model’s biggest strength is its simplicity in 


application. 


Chapter IV discussed the USAF’s KC-X Next Generation Tanker Planes 
Program, the authors gave some background information about the Air Force’s KC-X 
program. They used this program in their model to show the subjectivity involved in the 
source selection proposal evaluation part of the Air Force’s acquisitions. Introductory 
information about the program, the requirements of the program, the competition 
between the two offerors, and lastly some specific information (to roughly compare the 


two alternatives) about each proposed aircraft were presented in this chapter. 


In the KC-X Tanker Replacement Program, which is recently the most significant 
procurement program in the Air Force, the Air Force used its current source selection 
method, namely the color rating method, and awarded the contract to the Airbus/Northrop 
Grumman team. After debriefings were presented, the fact emerged that the ratings of 


both offerors were almost the same. Consequently, long debates have begun between 
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Boeing and the Air Force with Boeing filing a protest with the Government 
Accountability Office (GAO) for review of the award decision. The similar color ratings 
led to the protest exercised in the KC-X program and in some previous contract awards; 


therefore in the authors’ model this was one of the focus areas. 


In Chapter V about model construction, two models that were built within 
Microsoft Excel were presented in two versions to show the different characteristics of 
two types of methods that can be used in the source selection phase in government 
acquisitions. These methods are color rating, which is the current method used by the 
U.S. Air Force, and the quantitative weighted sum method. The models were established 
using some data from the Air Force’s current KC-X Tanker Replacement Program and 
proposals of its two offerors, Boeing and Northrop/Airbus. In Chapter VI that discusses 
the analysis and results, simplified KC-X program data was implemented in these 
models. Two offerors’ color rating data that are provided from publicly available sources 


were used in the models. 


In Chapter VII that presents the sensitivity analysis, the authors investigated how 
modeling choices might affect the results that were found in the previous chapter. A 
sensitivity analysis was done to understand the effects and limits of subjectivity in the 
authors’ models. Three different forms of sensitivity analysis were done. In the first two, 
the numerical ratings for the color and adjectival ratings were changed within their ranges 
to determine the best/worst possible overall scores for each alternative in the first and 
second versions of the authors’ model. In the third one, the weights were changed, while 
keeping original model ratings as constant, to analyze the effects of weights on the 


results. 
B. CONCLUSIONS 


In the authors' study, with their arbitrary numerical ratings chosen to be within the 
ranges of the color ratings given in the KC-X program, they were able to show that when 
they used the USAF’s color rating method, Airbus could have won the contract, but when 
they used the WSM method, Boeing could have won the contract. The implication under 


these results is that when there are small but important differences between offerors, the 
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color rating method is potentially incompetent to reflect those differences. However, the 
quantitative weighted sum method is capable of handling those little details and providing 


more accurate results. 


As explained previously, in the authors' models, version two differed from version 
one based on some changes to the color/adjectival ratings used and their numerical 
ranges. Initially, the color/adjectival ratings that reflected the lowest values for 
factors/subfactors were eliminated and then, the numerical ranges to which the color and 
adjectival ratings were related to were extended or narrowed. The arbitrarily chosen 
numerical ratings were then adjusted so that they still fell within the ranges of the color 
ratings given within the KC-X program. The second version of the authors' model, which 
suggested a different way to evaluate both offerors, produced the same significant result 


as was obtained in the first version. 


This part of the study supported the authors' argument that the quantitative 
weighted sum method is more competent than the color rating method in reflecting the 


little but important differences between offerors. 


In Chapter VII concerning sensitivity analysis, first the authors investigated how 
modeling choices might affect previous results that they found. They performed a 
sensitivity analysis to answer the research question: “How sensitive are the overall WSM 
scores to different numerical ratings within each color range?” While keeping the color 
rating ranges constant, the authors varied the numerical ratings within the given ranges 
for the color rating that was assigned to see how much better or worse the overall score 


could be while still corresponding to the same color rating. 


The sensitivity of overall scores for both alternatives was very similar. This is 
because while using color and adjectival rating method for evaluation, the scores for two 
alternatives were very similar. There were just three differences - one in the IFARA, one 
in the LCC, and one in cost risk rating. Only three out of the thirteen criteria that were 
affecting the overall score were different, albeit only slightly different. In addition, since 
these three factors were also multiplied by small weights, they made an even smaller 


difference in the overall score. Another important finding of these results is the high 
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percentage of overlap between the ranges of possible final scores for both alternatives. 
This overlapping area means that it is possible for either alternative to win the contract 


whenever the overall scores for both are in this overlapping range. 


Secondly, the sensitivity analysis used in the first part was used in the second 
model as well. The authors examined the results of the analysis to understand the effects 
of the choice of color rating ranges on the overall scores. The research question here was 
“How sensitive is the overall numerical score to different numerical ratings within each 
color range with a diminished number of color and adjectival ratings with appropriately 


adjusted color rating ranges?” 


Similarly with the first sensitivity analysis, the overall score sensitivities for both 
alternatives were very similar. It is because while using the color and adjectival rating 
method for evaluation, the scores for the two alternatives were very similar and also with 
the multiplier effect of weights these differences were getting less significant. Another 
important finding of these results is the high percentage of overlap between the ranges of 
possible final scores for both alternatives. This overlapping area means that it is possible 
for either alternative to win the contract whenever the overall scores for both are in this 
overlapping range. Logically, fewer ranges (rating groups) should cause a larger overlap 
region, supporting the notion that the less sensitive the method, the more likely it could 
undervalue small differences. Since the authors’ second version of their model had less 
number of color and adjectival ratings, the evaluation results became even less sensitive 


to the small differences. 


The sensitivity analysis to understand the effects of rating change within the color 
range on both versions of the authors’ study produced very similar results, namely a high 
percentage of overlap for final score ranges for both alternatives in both versions. The 
overlapping area means that it is possible for both alternatives to win the contract 
whenever the overall scores for both are in this overlapping range. The overlapping 
areabrings about the subjectivity issue to the evaluation of offeror proposals, because as 
the ranges got bigger in the second version, the effects of the rating change got bigger as 
expected and the subjectivity (with a growing overlapping range) involved increased as 


well. 
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In the third part of the authors’ sensitivity analysis, the weights were changed in 
both the color rating method and the WSM method, while the original model ratings were 
kept constant, to analyze the effects of weights on the results. In this analysis part, all the 
weights (for factors and subfactors) were varied to analyze the effects on overall scores. 
The sensitivity analysis to understand the effects of weight changes (including all factor 
and subfactor weights) on the overall scores produced interesting results. The range of 
changes for overall scores was very similar for Boeing and for Airbus in the color rating 
method. However, in the WSM method the change range difference was significant. 
These results were more substantiating data for the authors’ study to show how the color 
rating method is less sensitive than the WSM method (less competent to reflect the small, 


but potentially important differences between alternatives). 


In the last part of the sensitivity analysis, the authors also wanted to analyze the 
overall scores for two special cases. First, they solved for the weights that would give 
Boeing the best possible overall score. Then, they implemented Boeing’s best case 
scenario weights to calculate the overall score for Airbus in the same scenario. For the 
color rating method, Airbus won even with Boeing’s best scenario weights. For the WSM 
rating, the results were significantly better for Boeing. The authors also solved for 
Airbus’s best case scenario weights too and used them to calculate Boeing’s overall score 
in the same scenario. The results were very similar to the first special case, as Airbus won 
in with the color rating method even with a significant advantage. However, for the WSM 


method Boeing was far better than Airbus. 


It is necessary to remember that in this study, the authors had only the color 
ratings that were available, and therefore, they converted these color ratings to numbers 
and arbitrarily chose the ratings used in the WSM model to fall within the same color 
rating ranges. All the results came from the models where the authors used those arbitrary 
numbers. The purpose of the study was not to say who should have won the KC-X 
contract. The actual evaluation of proposals was done by a team consisting of many 
experts and the evaluation period was almost a year even for such a big team (the authors 


did not have enough information, time, and intent to do the same evaluation in a better 
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way). The authors simply wanted to show that the contract resulted in a different winner 
by using numbers and the WSM method instead of the color and adjectival ratings 


method. 


The color rating method was not sensitive enough for little differences in the 
evaluation ratings. These little differences in each factor or subfactor ratings may be in 
favor of one alternative. Thus, by using the color rating method one may eliminate all 
those little changes and consider both alternatives with the same color ratings and then 


possibly choose the wrong one (not the one that gives the best value). 


The authors’ sensitivity analysis also showed that the color rating range 
determination is very critical because the effects of these ranges on the results are pretty 
significant. The effects of color rating ranges are much higher than the effects of the 
weights of factors and subfactors. Additionally, the first (color rating range analysis on 
version one) and second sensitivity (same analysis on version two) showed the authors 
that with the less number of ratings they increased the ranges and as a result they made 


the color rating method even less sensitive to little differences. 


The third part of the sensitivity analysis results showed the effects of weights on 
overall scores. As mentioned before, the effects of weights were not as big as the effects 
of the color and adjectival rating ranges. One of the biggest reasons for the weights being 
less effective was because the verbal explanations of the relationships between these 
weights were given precisely in the RFP, so there was not much flexibility to change the 


weights. 


In conclusion, although the color rating method is currently the method for the Air 
Force, it is not necessarily competent whenever the scores of the proposed options are 
close. In some cases like the one the authors analyzed, the color rating method may 
eliminate some of the small differences (however, when aggregated these small 
differences may affect the overall score significantly) and may lead the decision maker in 
the wrong direction. The government has different responsibilities than the private sector, 
like being fair all the time. So, when seeking the best value, the government should make 


a fair evaluation of the proposals and explain the logic of their decision. With the color 
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rating method, the authors think that the government in some cases may not be able to 
make the evaluation fairly. This will not only lead to some trust issues but also lead to 
protests from the losers. With the protests, even the most important programs can be 


suspended and a significant amount of money can be spent in resolving the issues. 
C; RECOMMENDATIONS AND FUTURE RESEARCH 


The government’s responsibility is not only managing its programs successfully 
from a business stand point; government also has some other responsibilities like socio- 
economic responsibilities. The government must act fairly in all contract actions and 
making a fair evaluation of competing proposals is one of these responsibilities. 
Therefore, the government must be consistent in its evaluation of the proposals process 
and precise while explaining its requirements and methods for the evaluations. The color 
rating method is not a very competent method to obtain the best value, but it is clear that 
it gives the evaluator some space for flexibility when determining the best value for the 
government. The color rating method may end up eliminating the individual evaluation 
differences and establish a consensus on the score for individual factors and subfactors, 
but it is not competent enough to be used as the main method to determine which offeror 
will be awarded the contract. Although it seems that the color rating method is less 
subjective since it eliminates individual evaluation differences, it may come up with 
subjective results when used as the main method for evaluation of the proposals because 
of the elimination of small differences and their aggregate effects to the overall score. 
When the color rating results are similar, which is very likely to happen, it is hard to 
justify the choice that has been made as a government. Whenever the offerors are not 
satisfied with the results, and especially when they feel the results did not show a clear 
winner (the lack of clarity in the evaluation), they are very likely to protest the award, 


which leads to suspension of the program until all the protests are resolved. 


Instead of using a purely numerical model such as the weighted sum method, the 
color rating method can be developed again with an increased number of color and 
adjectival ratings instead of four or five ratings in order to increase its sensitivity to small 


differences. A more extensive numerical study could be performed to develop a 
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recommendation for how many colors to use (how sensitive one wants the color rating 
method to small differences) and how to define the conditions under each given color 


(similar to what is in place now for the red-yellow-green-blue color scheme). 


Another study could be done to understand the relationship between the color 
rating method and protests. This study could explore whether the U.S. Air Force was 
being protested more or less before they started using the color rating method and also 
determine whether protests under the color rating method are more common when the 


competing proposals were given similar color ratings. 
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APPENDIX 


Example Snapshot of Version One Color Rating Spreadsheet 
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Example Snapshot of Version One Weighted Sum Method Spreadsheet 
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The Snapshot of Version One Color Rating Method Spreadsheet for Boeing 
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The Snapshot of Version One Color Rating Method Spreadsheet for Airbus 
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The Snapshot of Version One Weighted Sum Method Spreadsheet for Boeing 
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The Snapshot of Version One Weighted Sum Method Spreadsheet for Airbus 
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Ratings Using Both Methods in Version One 
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The Comparison Chart Showing Both Offerors 
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The Snapshot of Version Two Color Rating Method Spreadsheet for Boeing 
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The Snapshot of Version Two Color Rating Method Spreadsheet for Airbus 
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The Snapshot of Version Two Weighted Sum Method Spreadsheet for Boeing 
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The Snapshot of Version Two Weighted Sum Method Spreadsheet for Airbus 
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The Comparison Chart Showing Both Offerors’ Ratings Using Both Methods in Version Two 
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Sensitivity Analysis Model for Best/Worst Ratings within Color Ranges 
(For Boeing in Version 1) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) rating. 
X2 : MC subfactor System Integration and Software (SIS) rating. 

x3 : MC subfactor Product Support (PS) rating. 

x4 : MC subfactor Program Management (PM) rating. 

x5: MC subfactor Technology Maturity and Demonstration (TMD) rating. 
X6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) rating. 
x7:  PResubfactor System Integration and Software (SIS) rating. 

xg :  PRsubfactor Product Support (PS) rating. 

x9: PRsubfactor Program Management (PM) rating. 

x10 :  Cost/Price subfactor Cost Risk (CR) rating. 

x7, : Past Performance (PP) rating. 


Objective Function (Linear Model): 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x44+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 x77 + 0.125 * 83.2 + 





(0.40 x19 + 0.60 * 39.12) 0.125 


Constraints: 


388 OIE 24S 88 IE 2S 8 OIE 2S 8 OIE 24S 18 OIE 2S 18 OIE IR 88 OIE IR 8 OIE 2S 88 OIE OIE 8 OIE 2 8 OIE OIE 8 IE 2S 8 OIE 2S 8 OIE 2 8 OIE Ie 8 2A 2g 28 2 2g 28 2S 2g 2g 2A 2g 2s 2A 2k 2g 2K 2k 2 2 2k 2s 2K 2K os ok 


For Mission Capability; 


x; >= 76 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 51 and x2 <=75 (Subfactor SIS is within green color rating range) 

x3 >= 76 and x3 <=100 (Subfactor PS is within blue color rating range) 

x4 >= 51 and x4 <=75 (Subfactor PM is within green color rating range) 

X5 >= 67 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 OIE 24S 8 2A 24g 8 88 2S 2 88 OIE 24S 18 OIE 24 1S OIE OIE 8 OIE OIE 18 OIE OIE 8 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE 2S 8 OIE 2g 8 OIE 2g 8 2A 2g 28 2k 2S 28 2A 2k 2 2K 2k 2g 2K 2k 2s 2 2k 2s 2K 2K oe ok 
For Proposal Risk; 

X6 >= 67 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 34 and x7 <=66 (Subfactor SIS is within moderate adjectival rating range) 
Xg >= 67 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 67 and x9 <=100 (Subfactor PM is within low adjectival rating range) 

388 IE 24S 8 FAS 24 8 88 2S IR 88 OIE 24S 18 OIE IE IS OIE 24 8 OIE OIE 8 OIE OIE 18 OIE OIE 18 FIC OIE 18 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE 2S 28 OIE 2g 88 OIE 2g 8 OIE 2g 2g 2 2g 2g 2k 2k 2 2K 2k 2g 2K 2k 2s 2K 2K ik ok 
For Cost/Price; 

x10 >= 34 and x9 <=66 (CR is within moderate adjectival rating range) 


BS OIE AR 88 2S 2S 88 OIE 2S 18 OIE 2S 18 OIE 24S 88 OIE 24S 18 OIE 2S 18 OIE 2A 18 OIE 2S 18 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE OIE 88 OIE OIE 8 OIE 2S 8 OIE 2g 88 OIE 2g 8 2S 2g 28 OI 2g 28 2A 2g 2s 2K 2k 2 2K 2k 2g 2K 2 is 2K 2k 


For Past Performance; 
X71; >= 51 and x;; <=66 (PP is within satisfactory adjectival rating range) 
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Sensitivity Analysis Model for Best/Worst Ratings within Color Ranges 
(For Airbus in Version 1) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) rating. 
X2 : MC subfactor System Integration and Software (SIS) rating. 

x3 : MC subfactor Product Support (PS) rating. 

x4 : MC subfactor Program Management (PM) rating. 

x5: MC subfactor Technology Maturity and Demonstration (TMD) rating. 
X6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) rating. 
x7:  PResubfactor System Integration and Software (SIS) rating. 

xg :  PRsubfactor Product Support (PS) rating. 

x9: PRsubfactor Program Management (PM) rating. 

x10 :  Cost/Price subfactor Cost Risk (CR) rating. 

x7, : Past Performance (PP) rating. 


Objective Function (Linear Model): 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x44+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 x7,;+ 0.125 * + 





(0.40 x70 + 0.60 * 39.8) 0.125 


Constraints: 


388 OIE 24S 88 IE 2S 8 OIE 2S 8 OIE 24S 18 OIE 2S 18 OIE IR 88 OIE IR 8 OIE 2S 88 OIE OIE 8 OIE 2 8 OIE OIE 8 IE 2S 8 OIE 2S 8 OIE 2 8 OIE Ie 8 2A 2g 28 2 2g 28 2S 2g 2g 2A 2g 2s 2A 2k 2g 2K 2k 2 2 2k 2s 2K 2K os ok 


For Mission Capability; 


x; >= 76 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 51 and x2 <=75 (Subfactor SIS is within green color rating range) 

x3 >= 76 and x3 <=100 (Subfactor PS is within blue color rating range) 

x4 >= 51 and x4 <=75 (Subfactor PM is within green color rating range) 

X5 >= 67 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 OIE 24S 8 2A 24g 8 88 2S 2 88 OIE 24S 18 OIE 24 1S OIE OIE 8 OIE OIE 18 OIE OIE 8 OIE 2S 18 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE 2S 8 OIE 2g 8 OIE 2g 8 2A 2g 28 2k 2S 28 2A 2k 2 2K 2k 2g 2K 2k 2s 2 2k 2s 2K 2K oe ok 
For Proposal Risk; 

X6 >= 67 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 34 and x7 <=66 (Subfactor SIS is within moderate adjectival rating range) 
Xg >= 67 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 67 and x9 <=100 (Subfactor PM is within low adjectival rating range) 

3S IE 24S 88 2A IR 8 18 OIE 2 8 OIE 2S 8 OIE 2A 18 FI 2A 18 OIE OIE 18 OIE 2S 18 OIE OIE 8 FIC IE 18 OIE OIE 18 OIE OIE 8 OIE IR 8 OIE 2S 18 OIE OIE 8 OIE 2S 8 OIE 2g 8 OIE 2g 8 2 2k 2 2k 2k 2g 2k 2k 2g 2K 2k is 2K 2K ie ok 
For Cost/Price; 

x10 >= 67 and x79 <=100 (CR is within low adjectival rating range) 


B88 IE 24S 8 2A 2S 88 OIE 2S 8 OIE 2S 8 OIE 2S 88 OIE IR 18 OIE 24S 18 OIE 24 18 OIE 2S 18 OIE OIE 8 OIE 2S 18 OIE 2S 8 OIE OIE 88 OIE IR 8 OIE 2S 8 OIE 2S 28 OIE 2g 8 2S 2g 28 2 2g 28 2k 2k 2g 2k 2k 2s 2K 2k 2g 2 2k is 2K 


For Past Performance; 
x7; >= 51 and x); <=66 (PP is within satisfactory adjectival rating range) 
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Sensitivity Analysis Model for Best/Worst Ratings within Color Ranges 
(For Boeing in Version 2) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) rating. 
X2 : MC subfactor System Integration and Software (SIS) rating. 

x3 : MC subfactor Product Support (PS) rating. 

x4 : MC subfactor Program Management (PM) rating. 

x5: MC subfactor Technology Maturity and Demonstration (TMD) rating. 
X6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) rating. 
x7:  PResubfactor System Integration and Software (SIS) rating. 

xg :  PRsubfactor Product Support (PS) rating. 

x9: PRsubfactor Program Management (PM) rating. 

x10 :  Cost/Price subfactor Cost Risk (CR) rating. 

x7, : Past Performance (PP) rating. 


Objective Function (Linear Model): 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x4+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 x7,;+ 0.125 * B+ 





(0.40 x19 + 0.60 * 39.12) 0.125 


Constraints: 


388 IE 24S 88 OIE 24S 88 2S 2S 8 OIE IR 18 OIE 2S 8 OIE 24 18 OIE 2A 8 OIE 24S 18 OIE OIE 18 OIE OIE 18 OIE OIE 8 OIE 2S 88 OIE 2S 8 OIE OIE 8 OIE 2g 8 2A 2S 28 2S 2g 28 2S 2g 28 2 2g 2 2K 2k 2s 2K 2k 2s 2 2 2s 2K 2K os ok 


For Mission Capability; 


x; >= 66 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 26 and x2 <=65 (Subfactor SIS is within green color rating range) 

x3 >= 66 and x3 <=100 (Subfactor PS is within blue color rating range) 

X4 >= 26 and x4 <=65 (Subfactor PM is within green color rating range) 

X5 >= 61 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 OIE 24S 8 2S 2S 8 88 2S 24S 18 OIE 24S 18 OIE IE 18 OIE IE 8 OIE OIE 18 OIE 24S 18 OIE 2S 18 OIE OIE 8 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2g 8 OIE 2g 8 OIE 2g 8 OIE 2S 28 2k 2g 8 2A 2k 2 2K 2g 2g 2K 2k 2 2K 2k 2s 2K 2K is ok 
For Proposal Risk; 

X6 >= 61 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 0 and x7 <=60 (Subfactor SIS is within moderate adjectival rating range) 
Xg >= 61 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 61 and x9 <=100 (Subfactor PM is within low adjectival rating range) 

38 OIE 2S 8 IS IR 8 18 OIE 24S 18 OIE 2S 18 OIE 2 18 OIE 24 1S OIE OIE 8 OIE IR 18 OIE OIE 18 OIE OIE 18 OIE OIE 18 OIE 2S 8 OIE OIE 8 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2g 8 OIE 2g 8 2K 2k 2g 2k 2k 2g 2k 2k 2g 2K 2k 2g 2K 2K ig ok 
For Cost/Price; 

x10 >= 0 and x 9 <=60 (CR is within moderate adjectival rating range) 


BS OIE 24S 88 2S 2S 88 OIE 2S 8 OIE IR 88 OIE 24S 18 2A 2A 88 OIE 2S 8 OIE IR 88 OIE 2S 18 FI 24 18 OIE OIE 8 OIE 2S 88 OIE fe 88 OIE 2S 18 OIE 2S 8 2S Ie 8 2A 2g 8 OIE 2k 28 OI 2g 28 2k 2k 2g 2A 2k 2s 2K 2k 2s 2K 2 is 2K 2k 


For Past Performance; 
x7; >= 31 and x;; <=60 (PP is within satisfactory adjectival rating range) 
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Sensitivity Analysis Model for Best/Worst Ratings within Color Ranges 
(For Airbus in Version 2) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) rating. 
X2 : MC subfactor System Integration and Software (SIS) rating. 

x3 : MC subfactor Product Support (PS) rating. 

x4 : MC subfactor Program Management (PM) rating. 

x5: MC subfactor Technology Maturity and Demonstration (TMD) rating. 
X6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) rating. 
x7:  PResubfactor System Integration and Software (SIS) rating. 

xg :  PRsubfactor Product Support (PS) rating. 

x9: PRsubfactor Program Management (PM) rating. 

x10 :  Cost/Price subfactor Cost Risk (CR) rating. 

x7, : Past Performance (PP) rating. 


Objective Function (Linear Model): 


Maximize Overall Score = (0.3 x; + 0.25 x2+ 0.2 x3+ 0.15 x4+ 0.1 x5) 0.25 + 
or (0.35 x6 + 0.3 x7 + 0.20 xg + 0.15 x9) 0.25 + 
Minimize 0.25 x7,;+ 0.125 * B+ 





(0.40 x70 + 0.60 * 39.8) 0.125 


Constraints: 


388 IE 24S 88 OIE 24S 88 2S 2S 8 OIE IR 18 OIE 2S 8 OIE 24 18 OIE 2A 8 OIE 24S 18 OIE OIE 18 OIE OIE 18 OIE OIE 8 OIE 2S 88 OIE 2S 8 OIE OIE 8 OIE 2g 8 2A 2S 28 2S 2g 28 2S 2g 28 2 2g 2 2K 2k 2s 2K 2k 2s 2 2 2s 2K 2K os ok 


For Mission Capability; 


x; >= 66 and x; <=100 (Subfactor KSR is within blue color rating range) 

X2 >= 26 and x2 <=65 (Subfactor SIS is within green color rating range) 

x3 >= 66 and x3 <=100 (Subfactor PS is within blue color rating range) 

X4 >= 26 and x4 <=65 (Subfactor PM is within green color rating range) 

X5 >= 61 and x5 <=100 (Subfactor TMD is within green color rating range) 

388 OIE 24S 8 2S 2S 8 88 2S 24S 18 OIE 24S 18 OIE IE 18 OIE IE 8 OIE OIE 18 OIE 24S 18 OIE 2S 18 OIE OIE 8 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2g 8 OIE 2g 8 OIE 2g 8 OIE 2S 28 2k 2g 8 2A 2k 2 2K 2g 2g 2K 2k 2 2K 2k 2s 2K 2K is ok 
For Proposal Risk; 

X6 >= 61 and x5 <=100 (Subfactor KSR is within low adjectival rating range) 

x7 >= 0 and x7 <=60 (Subfactor SIS is within moderate adjectival rating range) 
Xg >= 61 and xs <=100 (Subfactor PS is within low adjectival rating range) 

X9 >= 61 and x9 <=100 (Subfactor PM is within low adjectival rating range) 

388 IE IR 8 2S 24S 8 8 OIE 24 88 OIE 2S 8 OIE 2A IS OIE IE 8 OIE OIE 8 OIE 2S 18 OIE OIE 18 2K OIE 18 OIE OIE 18 OIE 2S 8 OIE 2 8 OIE 2S 8 OIE 2S 8 OIE 2S 8 OIE 2g 8 OIE 2g 8 2 2k 2 2k 2k 2g 2K 2k 2s 2K 2k 2s 2K 2K ig ok 
For Cost/Price; 

x10 >= 61 and x;9 <=100 (CR is within low adjectival rating range) 


BS OIE 24S 8 2S 24S 88 IE 2S 8 2S 2S 8 OIE 2S 88 OIE 2S 18 OIE 2S 8 OIE IR 18 OIE fe 18 OIE 2S 8 OIE OIE 8 OIE 2S 8 OIE 2 88 OIE 2S 8 OIE OIE 18 OIE 2g 28 OIE 2g 8 OIE 2g 8 2 2g 8 2k 2g 2g 2K 2k 2s 2K 2k 2s 2 2 is 2K 


For Past Performance; 
X71 >= 31 and x;; <=60 (PP is within satisfactory adjectival rating range) 
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Sensitivity Analysis Model for Best/Worst Ratings for Weight Changes 
(For Boeing for Color Rating Method in Version 3) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) weight. 
x2 : MC subfactor System Integration and Software (SIS) weight. 

x3: MC subfactor Product Support (PS) weight. 

xz : MC subfactor Program Management (PM) weight. 

xs: MC subfactor Technology Maturity and Demonstration (TMD) weight. 
x6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) weight. 
x7:  PRsubfactor System Integration and Software (SIS) weight. 

xs :  PRsubfactor Product Support (PS) weight. 

x9:  PRsubfactor Program Management (PM) weight. 

X10 :  Cost/Price subfactor Life Cycle Cost (LCC) weight. 

x1, :  Cost/Price subfactor Cost Risk (CR) weight. 

X72: Mission Capability, Proposal Risk and Past Performance factors’ weight. 
x;3 :  Cost/Price and IFARA factors’ weight. 


Objective Function (Non-linear Model): 





Maximize Overall Score =[(87.5 x;+ 62.5 x2+ 87.5 x3+ 62.5 x4+ 83.3 x5)] Xj. + 


or [(83.3 x6 + 50 x7 + 83.3 xg + 83.3 x9)] X12 + 
Minimize 58.35 X72 + 83.2 X73+ (50 Xi + 39.12 X10) X13 
Constraints: 


8 24S fe 8 88 588 2S 2S IR 2S 24S 24S 2S 8 88 2S 2S 2 2S 2S 2S 26 fe 88 2S OIE 2S 2K 2S IS 24g 2 8 218 IS 2S 2 IC 2S 24g fe 2 88 2S 2S 2S 2 2g fg fe 2 8 218 2S 2 2 2k 2g fe fe 2s 288 2g 2 2K 2K 2 2k 2 os 2k 


For Mission Capability; 





Xx) > X2 (KSR is more important than SIS) 

X2 > X3 (SIS is more important than PS) 

X3>X4 (PS is more important than PM) 

X4 > X5 (PM is more important than TMD) 

x5>0 (all of the weights are bigger than 0) 

Xp tx. t+ x34+X4+X5= 1 (total of weights are equal to 1) 
oe 2 of 2 of of of oe oe of oft of of oie oie aie oie ofe ofc oft oft of fe ofe ofc oft oft 2 of of ofc oie oe of of of ofc ofc ate ate afk ofe ofc oft fe ate afk ofc ofc of oft of of ofc oe oft of of of of oe oie oie afk ofc ofc ois oe oie oe of of of of ok ok ok 
For Proposal Risk; 

X6 > X7 (KSR is more important than SIS) 

X7 > Xg (SIS is more important than PS) 

Xg > X9 (PS is more important than PM) 

x9 >0 (all of the weights are bigger than 0) 

Xg + x7 + xg t+ Xo = 1 (total of weights are equal to 1) 
oie 2k ie fe fe fe of ofc of oft of af of ofc of of of of of oe oe oe of of of ois oie ate afk ofe ofc oft of aie afk ofe ofc oft of of of ofc of oie of of 2k oie ofc aie ate ofe ofc ofc of aie afk of ofc ofc oft of of of fe oie of of of 2k ok oie oe oe of of ok ok 
For Cost/Price; 

X10 > X11 (LCC is more important than CR) 

x11>0 (all of the weights are bigger than 0) 

Xo + x= 1 (total of weights are equal to 1) 


BES 8 28 IS IS OIE 24S IR 24S 8 IS 8 2S IS IS OIE OIE 24S 24S 2 8 24 8 8S IS IS OIE 2S OIE OIE 24S 2 24E 2 8 IS 2S IE IE OIE OIE 24S 248 OIE 24S 2 8 28S 2S 2S OIE OIE fe 24g 2k fe fe fe 8 288 2S 2S 2S 2k 2g 2g 2g fe fe fe is 2s 2g 2K 2 ke 


For Factor Level; 


X12 > X23 (MC,PR and PP are more important than CR and IFARA) 
X13>0 (all of the weights are bigger than 0) 
3.x2+2x73=1 (total of weights are equal to 1) 
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Sensitivity Analysis Model for Best/Worst Ratings for Weight Changes 
(For Airbus for Color Rating Method in Version 3) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) weight. 
x2 : MC subfactor System Integration and Software (SIS) weight. 

x3: MC subfactor Product Support (PS) weight. 

xz : MC subfactor Program Management (PM) weight. 

xs: MC subfactor Technology Maturity and Demonstration (TMD) weight. 
x6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) weight. 
x7:  PRsubfactor System Integration and Software (SIS) weight. 

xs :  PRsubfactor Product Support (PS) weight. 

x9:  PRsubfactor Program Management (PM) weight. 

X10 :  Cost/Price subfactor Life Cycle Cost (LCC) weight. 

x1, :  Cost/Price subfactor Cost Risk (CR) weight. 

X72: Mission Capability, Proposal Risk and Past Performance factors’ weight. 
x;3 :  Cost/Price and IFARA factors’ weight. 


Objective Function (Non-linear Model): 





Maximize Overall Score =[(87.5 x; + 62.5 x2 + 87.5 x3+ 62.5 x4+ 83.3 x5)] x12 + 


or [(83.3 x6 + 50 x7 + 83.3 xg + 83.3 x9)] X12 + 
Minimize 58.35 X12 + 92 X73+ (83.3 Xap + 39.8 X10) X13 
Constraints: 


8 24S 24 8 8 518 2S 2S 2 IR 24S 24S fe 8 88 IS OIC 2S 2 IR IS 2S 2 88 IS OIE 2S 2 2S 24g 246 2 8 288 IS 2S 2S IR 24S 24g 2S 8 8 2S 2S 2S 2 24k fg 246 2 88 28 2S 2 2 2k 2k fe fe 2 288 2 2 2K 2K 2 2k 2k os 2k 


For Mission Capability; 





Xx) > X2 (KSR is more important than SIS) 

X2 > X3 (SIS is more important than PS) 

X3>X4 (PS is more important than PM) 

X4 > X5 (PM is more important than TMD) 

x5>0 (all of the weights are bigger than 0) 

Xp tx. t+ x34+X4+X5= 1 (total of weights are equal to 1) 
oe 2 of 2 of of of oe oe of oft of of oie oie aie oie ofe ofc oft oft of fe ofe ofc oft oft 2 of of ofc oie oe of of of ofc ofc ate ate afk ofe ofc oft fe ate afk ofc ofc of oft of of ofc oe oft of of of of oe oie oie afk ofc ofc ois oe oie oe of of of of ok ok ok 
For Proposal Risk; 

X6 > X7 (KSR is more important than SIS) 

X7 > Xg (SIS is more important than PS) 

Xg > X9 (PS is more important than PM) 

x9 >0 (all of the weights are bigger than 0) 

Xg + x7 + xg t+ Xo = 1 (total of weights are equal to 1) 
oie 2k ie fe fe fe of ofc of oft of af of ofc of of of of of oe oe oe of of of ois oie ate afk ofe ofc oft of aie afk ofe ofc oft of of of ofc of oie of of 2k oie ofc aie ate ofe ofc ofc of aie afk of ofc ofc oft of of of fe oie of of of 2k ok oie oe oe of of ok ok 
For Cost/Price; 

X10 > X11 (LCC is more important than CR) 

x11>0 (all of the weights are bigger than 0) 

Xo + x= 1 (total of weights are equal to 1) 


BES 8 28 IS IS OIE 24S IR 24S 8 IS 8 2S IS IS OIE OIE 24S 24S 2 8 24 8 8S IS IS OIE 2S OIE OIE 24S 2 24E 2 8 IS 2S IE IE OIE OIE 24S 248 OIE 24S 2 8 28S 2S 2S OIE OIE fe 24g 2k fe fe fe 8 288 2S 2S 2S 2k 2g 2g 2g fe fe fe is 2s 2g 2K 2 ke 


For Factor Level; 


X12 > X23 (MC,PR and PP are more important than CR and IFARA) 
X13>0 (all of the weights are bigger than 0) 
3.x2+2x73=1 (total of weights are equal to 1) 
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Sensitivity Analysis Model for Best/Worst Ratings for Weight Changes 
(For Boeing for WSM Method in Version 3) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) weight. 
x2 : MC subfactor System Integration and Software (SIS) weight. 

x3: MC subfactor Product Support (PS) weight. 

xz : MC subfactor Program Management (PM) weight. 

xs: MC subfactor Technology Maturity and Demonstration (TMD) weight. 
x6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) weight. 
x7:  PRsubfactor System Integration and Software (SIS) weight. 

xs :  PRsubfactor Product Support (PS) weight. 

x9:  PRsubfactor Program Management (PM) weight. 

X10 :  Cost/Price subfactor Life Cycle Cost (LCC) weight. 

x1, :  Cost/Price subfactor Cost Risk (CR) weight. 

X72: Mission Capability, Proposal Risk and Past Performance factors’ weight. 
X;3 :  Cost/Price and IFARA factors’ weight. 


Objective Function (Non-linear Model): 





Maximize Overall Score =[(93 x; + 64 x. + 92 x3+ 63 x4+ 94 x5)| x2 + 








or [(94 x5 + 57 x7 +97 xg + 94 xX6)] X72 + 
Minimize 59 x10 + 83.2 x)3+ (58 x77 + 39.12 X10) X13 
Constraints: 


DS 24S 24 8 88 588 IS 2S 2 IR 24S 24S fe 8 88 2S 2S 2 2 IR 2g 248 2 18 2S OIE OIC 2 IR 2S 248 2 8 218 2S 2S 2 IR 2S 24g fg 8 218 2S 2S 2S IC 2k fg 246 2 8 2S 2S 2 2 2k 2k fe fe 2 288 2g 2 2K 2K 2K 2k 2 os 2k 


For Mission Capability; 





Xx) > X2 (KSR is more important than SIS) 

X2 > X3 (SIS is more important than PS) 

X3>X4 (PS is more important than PM) 

X4 > X5 (PM is more important than TMD) 

x5>0 (all of the weights are bigger than 0) 

Xp tx. t+ x34+X4+X5= 1 (total of weights are equal to 1) 
oe 2 of 2 of of of oe oe of oft of of oie oie aie oie ofe ofc oft oft of fe ofe ofc oft oft 2 of of ofc oie oe of of of ofc ofc ate ate afk ofe ofc oft fe ate afk ofc ofc of oft of of ofc oe oft of of of of oe oie oie afk ofc ofc ois oe oie oe of of of of ok ok ok 
For Proposal Risk; 

X6 > X7 (KSR is more important than SIS) 

X7 > Xg (SIS is more important than PS) 

Xg > X9 (PS is more important than PM) 

x9 >0 (all of the weights are bigger than 0) 

Xg + x7 + xg t+ Xo = 1 (total of weights are equal to 1) 
oie 2k ie fe fe fe of ofc of oft of af of ofc of of of of of oe oe oe of of of ois oie ate afk ofe ofc oft of aie afk ofe ofc oft of of of ofc of oie of of 2k oie ofc aie ate ofe ofc ofc of aie afk of ofc ofc oft of of of fe oie of of of 2k ok oie oe oe of of ok ok 
For Cost/Price; 

X10 > X11 (LCC is more important than CR) 

x11>0 (all of the weights are bigger than 0) 

Xo + x= 1 (total of weights are equal to 1) 


BES 8 28 IS IS OIE 24S IR 24S 8 IS 8 2S IS IS OIE OIE 24S 24S 2 8 24 8 8S IS IS OIE 2S OIE OIE 24S 2 24E 2 8 IS 2S IE IE OIE OIE 24S 248 OIE 24S 2 8 28S 2S 2S OIE OIE fe 24g 2k fe fe fe 8 288 2S 2S 2S 2k 2g 2g 2g fe fe fe is 2s 2g 2K 2 ke 


For Factor Level; 


X12 > X23 (MC,PR and PP are more important than CR and IFARA) 
X13>0 (all of the weights are bigger than 0) 
3.x2+2x73=1 (total of weights are equal to 1) 
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Sensitivity Analysis Model for Best/Worst Ratings for Weight Changes 
(For Airbus for WSM Method in Version 3) 


Decision Variables: 





x; : Mission Capability (MC) subfactor Key System Requirements (KSR) weight. 
x2 : MC subfactor System Integration and Software (SIS) weight. 

x3: MC subfactor Product Support (PS) weight. 

xz : MC subfactor Program Management (PM) weight. 

xs: MC subfactor Technology Maturity and Demonstration (TMD) weight. 
x6 : Proposal Risk (PR) subfactor Key System Requirements (KSR) weight. 
x7:  PRsubfactor System Integration and Software (SIS) weight. 

xs :  PRsubfactor Product Support (PS) weight. 

x9:  PRsubfactor Program Management (PM) weight. 

X10 :  Cost/Price subfactor Life Cycle Cost (LCC) weight. 

x1, :  Cost/Price subfactor Cost Risk (CR) weight. 

X72: Mission Capability, Proposal Risk and Past Performance factors’ weight. 
X;3 :  Cost/Price and IFARA factors’ weight. 


Objective Function (Non-linear Model): 





Maximize Overall Score =[(81 x;+ 60 x.+ 90 x3+ 58 x4+ 85 x5)] Xy2 + 








or [(85 x5 +57 x7 +91 xg + 87 X6)] X22 + 
Minimize 56 X12 + 92 X73+ (70 x11 + 39.8 X10) X13 
Constraints: 


DS 24S fe 8 88 218 2S 2S 2 2S 24S 246 fe 8 288 IS OIC 2S 2 OIE 24g 246 2 88 2S OIE IC 2 IR 2S 24g 2 8 218 OIE 2S 2 IR 2g 246 2S 8 8 2S 2S 2K IR 2k fg fe fe 88 28 2S 2S 2 2k fe fe fe 2s 28 2 2 2K 2 2 2k 2k 2s 2k 


For Mission Capability; 





Xx) > X2 (KSR is more important than SIS) 

X2 > X3 (SIS is more important than PS) 

X3>X4 (PS is more important than PM) 

X4 > X5 (PM is more important than TMD) 

x5>0 (all of the weights are bigger than 0) 

Xp tx. t+ x34+X4+X5= 1 (total of weights are equal to 1) 
oe 2 of 2 of of of oe oe of oft of of oie oie aie oie ofe ofc oft oft of fe ofe ofc oft oft 2 of of ofc oie oe of of of ofc ofc ate ate afk ofe ofc oft fe ate afk ofc ofc of oft of of ofc oe oft of of of of oe oie oie afk ofc ofc ois oe oie oe of of of of ok ok ok 
For Proposal Risk; 

X6 > X7 (KSR is more important than SIS) 

X7 > Xg (SIS is more important than PS) 

Xg > X9 (PS is more important than PM) 

x9 >0 (all of the weights are bigger than 0) 

Xg + x7 + xg t+ Xo = 1 (total of weights are equal to 1) 
oie 2k ie fe fe fe of ofc of oft of af of ofc of of of of of oe oe oe of of of ois oie ate afk ofe ofc oft of aie afk ofe ofc oft of of of ofc of oie of of 2k oie ofc aie ate ofe ofc ofc of aie afk of ofc ofc oft of of of fe oie of of of 2k ok oie oe oe of of ok ok 
For Cost/Price; 

X10 > X11 (LCC is more important than CR) 

x11>0 (all of the weights are bigger than 0) 

Xo + x= 1 (total of weights are equal to 1) 


BES 8 28 IS IS OIE 24S IR 24S 8 IS 8 2S IS IS OIE OIE 24S 24S 2 8 24 8 8S IS IS OIE 2S OIE OIE 24S 2 24E 2 8 IS 2S IE IE OIE OIE 24S 248 OIE 24S 2 8 28S 2S 2S OIE OIE fe 24g 2k fe fe fe 8 288 2S 2S 2S 2k 2g 2g 2g fe fe fe is 2s 2g 2K 2 ke 


For Factor Level; 


X12 > X23 (MC,PR and PP are more important than CR and IFARA) 
X13>0 (all of the weights are bigger than 0) 
3.x2+2x73=1 (total of weights are equal to 1) 
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Microsoft Excel Snapshot of Sensitivity Analysis Model for Best Rating within Color Ranges for Boeing in Version 1 





Al B e | B&B | £ i F£ Be Ue) ot i a. I eT 


PAST PERFORMANCE 
SATISFACTORY | 51-66 | 








MISSION CAPABILITY 
Mareens EHS mes 
fF KSR | 76-100 RTD 
| sis | (51-75 Wg 0.25 18.75 
| Ps | 76-100 BR _ RATING | 
PP | at76 | 179 | 83.20 
| TD | 67-100 RL 0.10 | 83.20 | 














[neon 


Mission Capability | 0.250| 90.00] __—_—22.60 
Proposal Risk | —0.260|89.80| 22.45 


WEIGHTED 
PROPOSAL RISK SUBFACTORS |WEIGHTS RATINGS 


KSR_|_ 67-100 RI | _36.00 
sis | 34-66 RI 0.30 | 19.80 
Ps | 67-100 RI 0.20 | 20.00 
Pu | 67-100 R015 | 16.00 


| cost | 108.04 | MA | 0.60 | 39.12, 23.47| 
| RISK 34-66 | 040 | 66.00/ 26.40) 








VERALL OFFEROR RATING: 


COSTIPRICE SUBFACTORS 
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Microsoft Excel Snapshot of Sensitivity Analysis Model for Best Rating within Color Ranges for Boeing in Version 2 




















PAST PERFORMANCE| RATING | 
60 60.00 
[ne eo] me Fae 
Mission Capability 0.250] 86.00 | — 21.50) 
Proposal Risk 0.250] 88.00 | — 22.00] 
Past Performance 0.250/ 60.00 | — 15.00] 
Cost/Price 0.125] 47.47 | = 5.93 
IFARA 0.125] 79.00 
VERALL OFFEROR RATING: 743 
COSTIPRICE 
suaractors |W" OnTS acs TN 
cosT | 108.044 | 060 | 39.12) 23.47 
| RISK R040 | 60.00] 24.00 
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Microsoft Excel Snapshot of Sensitivity Analysis Model for Best Rating for Weight Changes for Boeing 
for Color Rating Method In Version 3 



















(PAST PERFORMANCE | RATING _ 
[SATISFACTORY | 58.6 
KS [sue 
sis [ore fame 
WTS 00: 
Pi [oree fame 
[Two [erect 


PROPOSAL RISK 
sueracrone | WERHTS = FACTORS WEIGHTS 
is _|__ag 


| KSR [LOW | 0.94 Mission | Mission Capability 0.327 
| SIS [MODERATE IMIR 0.327 
| Ps [low a 0.327 
| PM Low ay Pas 0.010 


0.010 
1.000 


COSTIPRICE 
susractors | WEIGHTS we BOEING COLOR ORIGINAL 68.02 


ie 39.12 | 19.76 
| RISK [MODERATE MS 60 | 24.75) 


MISSION CAPABILITY 
7876 








| 179] 83.20) 
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Microsoft Excel Snapshot of Sensitivity Analysis Model for Best Rating for Weight Changes for Boeing 
for WSM Method in Version 3 








PROPOSAL RISK 
ae di oo | Facrors | kisi jews |e 
= Mission Capability 0.327 2 Be 
Proposal Risk 0.327 


oy 
o0t0 
re —— EE — eso) 0 

378 


1.000 


COSTIPRICE 
suractors | WEIGHTS = ae BOEING WSM ORIGINAL 72.15 
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CRsk | WA Om 2 [ 271 
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Sensitivity Analysis Boeing’s Best Case Scenario Weights in Color Rating Method 
Applied to Airbus 


BOEING AIRBUS 


MISSION MISSION 

EIGHT: WEIGHTED WEIGHTED 

CAPABILITY dll CAPABILITY welts ple 

| KSR [BLUE 0.90 78.75 | KSR_ [BLUE 0.90 78.75 
| SIS GREEN fmmIKIt | SIS GREEN famUK) 
| PS [BLUE 0.03 | PS [BLUE 0.03 
| PM |GREEN miki | PM |GREEN #amKiy 
| TMD |GREEN famKy | TMD |GREEN famKul 


PROPOSAL RISK wanes PROPOSAL RISK FIRES 
sears NOT ICS 
| KSR {LOW RS 83.3 | 78.30 | KSR [LOW tS 30 
| SIS |wooeraTe MUN 50 | 1.50 | SIS_|vooeRare RMI 
| PS (LOW ; | PS (LOW ih 
PPM LOW ea | PM LOW 

|__ 82.30) 


COSTPRICE COSTPRICE 
WEIGHT: WEIGHTED 
SUBFACTORS aca raTNs RATINGS SUBFACTORS elt RATINGS 
| 39.8 _| 


COST | 10804 MAN 39.12 | 19.76 Cost | 108.01 Gy 
RISK [oneraTe UR 50 | 24.75 
| 44.51) 


RATING | FARA | RATING RATING | TFARA | RATING 
SATISFACTORY [2825 [L179] e820) [SATISFACTORY | sea3]_19 | 9200 


| 58.35 _| | 83.20) | 58.35} | 92.00) 

WEIGHTED WEIGHTED 

= ses [wn] une 

0.327 0.327 
Proposal Risk Proposal Risk ,amURY) 
Past Performance #am's 0.327 
Cost/Price F 0.010 
0.010 
1.000 


BOEING COLOR ORIGINAL 68.019 AIRBUS COLOR ORIGINAL 70.835 
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Sensitivity Analysis Airbus’s Best Case Scenario Weights in Color Rating Method 
Applied to Boeing 


AIRBUS BOEING 


MISSION MISSION 
WEIGHT: fees WEIGHT Seer) 


KSA Lue KsA_ BLUE — eT 
sis GREEN fae -— 250) [sis [GREEN fame 
ise ars a) [pea 
Pw GREEN fea 625 | 125] | pM [GREEN fe 
adr a0 


| KSR [LOW ie | KSR_ (LOW 
| SIS [woperare BIO 50} | SIS [Moperare BMMUO 
| PS [LOW ty 67} | PS [LOW ty 
LPM [LOW ty a] 0.01 


COSTIPRICE wai COST/PRICE wai 
WEIGHT 

SUBFACTORS pe suBFActors | V=/GHTS RATINGS ao 

COST | 10801 Mam 39.8 | 20.10] | COST | 108044 MNa—NN 39.12 | 19.70 

LRISK | LOW FR 83.3 iz |RISK_|MobeRATE R50 a 



















PAST PERFORMANCI RATING | IFARA | RATING 
SC 


WEIGHTED WEIGHTED 
FACTORS RATING FACTORS RATING 


| Mission Capability JURA 85.96 | Mission Capability MUP = 85.96 
Proposal Risk famUPAIy! 3307 | 8d Proposal Risk  famU Rat! 


PAST PERFORMANCI RATING | IFARA | RATING 
SATISFACTORY een 


A 58.35 | 190 Past Performance [amURAl!! 
REE 61.33 11.90 Cost/Price 0.194 
Fam — ERE 00] 75] [rae — et 


1.000 97 1.000 





AIRBUS COLOR ORIGINAL 70.835 BOEING COLOR ORIGINAL 68.019 
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Sensitivity Analysis 1 Results Summary Table 


VERSION 1 | BOEING | AIRBUS | DIFFERENCE [OVERLAPPING RANGES | 





MAX | 78.084 | 80.935 | 2851 16.949 
MIN | 58.334] 61135 | 2001 _| | 


19.75 19.8 


Sensitivity Analysis 2 Results Summary Table 


VERSION 2_ | BOEING | AIRBUS | DIFFERENCE | OVERLAPPING RANGES 


MIN — | = 43.609 48.085 4.476 : 


30.7 29.65 





Sensitivity Analysis 3 Results Summary Table 


COLOR MAX COLOR MIN | WSMMAX 
75.30 r 65.04. —~*Y| 80.75 67.38 
ORIGINAL 68.02 ORIGINAL 72.15 
75.97 | 66.95 73.44 66.18 
eee ORIGINAL 70.84 ORIGINAL 70.14 


COLOR WEIGHT RANGE WSM WEIGHT RANGE 
BOENG | ———SSC~—~SSCSC~<“—~*~sSBSSS 
AIRBUS_|902.—SCSCS<‘<‘<7RPTSC<CS~S*~‘S SCS 


BOEING'S BEST RESULT RATINGS VS. AIRBUS WITH SAME WEIGHTS 





BOEING _| 
AIRBUS 
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